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ABSTRACT 

This document reports the Northern Territory Primary 
Assessment Program (Australia), which monitors student achievement in 
reading and mathematics in grades 5 and 7 in urban primary schools 
and grade 5 to post-primary in Aboriginal schools. The aim of the 
program is to provide teachers with a set of assessment materials and 
provide information regarding the academic achievement of students at 
the end of primary schooling. This study focused on the operation of 
the program in schools and the perceptions of teachers and other 
personnel involved with the program. Data were collected by a 
questionnaire-survey of 185 teachers and school visits to 19 urban 
schools and 10 rural Aboriginal schools. Results of the study 
indicate the following: (1) many teachers have little understanding 
or knowledge of the program; (2) some schools have incomplete sets of 
materials; (3) teachers expressed doubts about the relevance of 
certain program materials for classroom use; and (4) only about half 
of all Aboriginal schools participate in the program. In schools 
where the program has been properly implemented it makes a valuable 
contribution; therefore, active participation must be ensured for all 
schools to take advantage of the program's potential. Appendices 
include procedures for improving test item quality, interview 
schedules, sample questionnaires for urban and Aboriginal schools, 
and a sample program evaluation form. (LP) 
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FOREWORD 



This is an interesting study. Essentially it brings togetlier 
information from the documentary record of the Primary 
Assessment Rrogram and samples perceptions of the program by 
those working in schools. It is not a full-scale evaluation of the 
program because the data are derived mainly from these two 
sources and the study has not encompassed other perspectives 
which would give a more complete picture. 



In summary, the results of this study show that in those schools 
where the program has been properly implemented the program 
makes a valuable contribution. It also shows a widespread lack 
of appreciation of the program and the advantages it offers for 
good teaching. The challenge is obvious. The program makes 
available to schools high quality and relevant assessment 
materials and establishes criteria for judging progress. We must 
ensure that all schools take advantage of its potential. 



It should be remembered that teachers and principals have been 
closely involved with the development of the Primary 
Assessment Program from its inception. Changes to the program 
have largely been in response to demands from schools. 
Considerable efforts have been made to provide in-service and 
support. -In these circumstances, it is difficult to understand the 
resistance the program sometimes still evokes, particularly as 
none of the dire predictions, that some made at its beginning, 
have come about. 



All schools are required to participate in the program and I look 
forward to their doing so, knowledgeably and enthusiastically. 
We will do what we can by way of in-service and support to 
ensure staff are awa-e of what the program offers, and I hop(=^ all 
schools will not only take advantage of these opportunities but 
also see it as a partnership in which they can do much to help 
themselves. Very considerable resources have gone into this 
program and some schools are deriving great benefits. These are 
available to .all. 



The Primary Assessment Program, particularly for urban schools, 
is reaching maturity. The fears that accompanied its birth have 
not been realised. Prejudice against it has no real foundation. 
Under adverse circumstances it has proven itself and the time is 
— -ripe for its full advantages to- be exploited by all. - - 



Dr C H Payne 
Director 

Curriculum & Assessment Branch 
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1.0 INTRODUCTION 



1.1 Reason for the Evaluation 

Improving academic standards is a long standing concern of 
educationists and can be a catalyst for changes in educational 
practice. One of the methods of addressing this concern in the 
Northern Territory was the introduction of Core Curricula into 
schools in 1981. Results of NT students in the 1980 Australian 
Studies in Student Performance Project tests (Wee, 1982) provided 
impetus for a system-wide assessment of core objectives. A 
working party was set up in 1982 to develop procedures for 
assessing student performance at the primary school level. This 
resulted in the Northern Territory Primary Assessment Program 
which was introduced into urban schools in 1984 and Aboriginal 
schools in 1986. 



This study of the Primary Assessment Program originated from a 
motion passed by the Board of Studies in 1987 "That the NT 
Principals Association be requested to survey- its members as to the 
value they perceived in the Primary Assessment Program" (Minutes 
of the Board of Studies meeting 87/4). Subseauently, as a preferred 
strategy, the Evaluation, Research and Accreditation Section of the 
Curriculum and Assessment Branch was asked to carry out the study. 
As the assessment program affects a wider audience than 
principals, the study was extended to include the views of teachers 
and other Departmental personnel involved with the program. 

In accordance with the original request from the Board of Studies 
the emphasis has been upon the way the program has been received 
and implemented in schools. Thus it is not a full-scale evaluation of 
the program. For example, the study did not extend to any in-depth 
or expert appraisal of the intrinsic merit of the materials or test 
items. The test items have, however, been subjected to statistical 
checks for .validity and reliability as part of the ongoing 
development of the program (Appendix I. Procedures for Improving 
Item Quality. Wee, 1986, pp, 4 - 6). 

This report summarises the findings of the study carried out during 
1988/89. It focuses mainly on the operation of the program in 
schools and relies primarily on the perceptions and opinions of 
school staff for the main body of information. 



1.2 What is the Primary Assessment Program? 



The NT Primary Assessment Program is an assessment package 
employed - -in. the. Northern Territory., to. assess .competencies in 
Mathematics and English of children in Years 5 and 7 in urban 
primary schools and Year 5 to post-primary in Aboriginal schools. 

The main aims of the Primary Assessment Program are; to provide 
teachers in schools with a set of assessment materials, and to 
provide the Department (and the community) with information 
regarding the academic standards of pupils at the end of primary 
schooling. To this end the program has two facets: 

(a) A set of assessment materials based on the Mathematics and 
English curricula which have been sent to cschools for teachers to 
use in assessing a range of knowledge and skills in these subjects. 

b) Sets of tests, based on the assessment materials, that are 
administered to pupils in Year 5 and Year 7 in urban schools and Year 
5 to post-primary in Aboriginal schools at the end of each year, with 
results forwarded to the Department to provide system-wide data 
on academic standards across the Territory and class results for 
schools. 

A more detailed description of the Primary Assessment Program is 
provided in Chapter 3. 

* 

1.3 Origins and Implementation of the Program 

In the early 1980's there were widespread moves throughout 
Australia for greater accountability in Education and increased 
recognition of the need for evaluation and assessment. Tliis was 
reflected in a comment made by the then chairman of the Northern 
Territory Curriculum Advisory Committee: 

....Increasingly the State systems will be held responsible 
for educational outcomes by parents, students, employers 
and Federal and State politicians. ...Of one thing I am 
firmly convinced. We in education today must be willing 
to evaluate and assess what we are doing, and be guided 
by the results. If we do not; someone else will do it for 
us (Chari man's Report, Curriculum Advisory Committee - 
Meeting 81/2, 1981). 
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The following statement from the paper "Enhancing Standards in 
Northern Territory Primary Schools" (1988) captures the 
circumstances surrounding the origins of the Primary Assessment 
Program: 



A public debate on standards was triggered in 1982 by a 
Ministerial statement on systemwide testing wtiicti 
foiiowed the release of ttie results of Northern Territory 
students in the Australian Studies in Student Performance 
Project (ASSP).... A working party was set up in November 
• 1982 by the then {Minister of Education to formulate a 
proposal for a primary assessm.ent program (Huang, 1987, 

p. D- 

Work on the development of materials began in 1983 with the first 
set of materials being sent to urban schools at the beginning of 
1984. A series of inservice workshops were held to explain the 
program. Tests taken from the materials were administered to 
students at the end of the year and the first report on the results 
was published in 1985, 

In 1986, materials developed for Aboriginal schools were sent to 
schools along with instructions for their use. In September of that 
year schools were asked to administer specified tests from the 
materials and send results to the Department. Tho first report on 
the results from Aboriginal schools was published in 1987. 

Since 1986 additional assessment materials have been developed and 
sent to schools. All Aboriginal and urban schools are expected to 
administer tests annually. 

Major reports and summary reports based on tost results are written 
each year. These reports are presented to the Board of Studies and if 
endorsed then go to the Minister for Education through the Secretary. 
The summary reports are then released to the general public when 
this process is complete. 



2 0 METHODOLOGY 



2.1 Objectives of the Study 



As a guide for the study the following objectives were formulated: 

(i) report how the program operates at school and 
Departmental level; 

(ii) gauge teacher awareness and use of materials; 
(ill) gauge teacher attitudes towards the program; 

(iv) determine the strengths and weaknesses of the program as 
perceived by personnel involved; 

(v) describe materials and reports; 

(vi) report the development and implementation of the program 
including any changes; and 

(vii) report the use made of information gained from the 
testing program. 



2.2 Data Collection 

2.2.1 Overview 

To study as many aspects of the program as possible, a wide range of 
data was required. Activities included: searching literature to 
provide an historical perspective on the Northern Territory program; 
examining materials, test results and reports; surveying opinion by 
questionnaire and interview; and observing the operation of the 
program at school and Departmental levels. 

2.2.2 Structured Interview 

The interview schedule developed for measuring levels of use of an 
innovation (Loucks, Newlove & Hall, 1975) was adapted and 
employed. This schedule gauges the participants' awareness and use 
of an innovation. Additional topics based directly on the Northern 
Territory Primary Assessment Program wer« included. After 
trialling in four urban and two Aboriginal schools the interview 



schedule was revised to allow more flexibi'ity to cater for the range 
of familiarity participants had with materials and their experience 
with the program (Appendix II), 

2.2,3 Questionnaires 

Questionnaires, based on the objectives of the study were developed 
for: 



(a) principals and senior teachers in urban primary schools 
(Appendix 111); 

(b) teachers of Years 5, 6, and 7 in urban primary schools 
(Appendix IV); 

(c) principals/assistant principals and teachers in charge of 
Aboriginal schools (Appendix V); 

(d) teachers of upper primary and post-primary classes in 
Aboriginal schools (Appendix VI). 

The questionnaires and interview schedules were trialled in four 
urban schools and two Aboriginal schools. In the trials these 
instruments were administered personally by the evaluator, and 
teachers' comments on the questions, format, etc., noted. 

Because a new style of Mathematics test, based on the Board 
Approved Curriculum, was developed for Year 7 in 1988, an 
additional questionnaire in the form of a one page feedback sheet 
was developed in November of that year. This questionnaire was 
aimed at providing information on teachers' opinions as to the time 
allowed for students to complete the test, the date of the test, the 
range and difficulty of items and the test format (Appendix VII). 

2.2.4 Document and Materials Search 

Historical and current documents related to the program were 
studied. This included examining files, reports and relevant letters 
and circulars. 

Assessment materials developed for the program were examined and 
a brief description made of each. Some discussion was held with 
materials developers and changes that had been made to materials 
and the testing program were noted. 

Annual reports on test results were studied and a summary made of 



how the program operated each year with a brief outline of the 
general findings on academic attainment for the groups tested. 

2.2.5 Respondents and Informants 

Questionnaires, for the four groups identified in 2.2.3, were sent to 
66 urban primary schools and 77 Aboriginal schools in June 1988. 
For the larger urban schools where there was a possibility of more 
than one teacher for each of the grades 5, 6, and 7 two 
questionnaires were sent for each year level and two questionnaires 
for principals and/or seniors to complete. For the smaller schools 
one questionnaire was sent for each of the year levels and two 
questionnaires for the principals and/or senior teachers. Because of 
a low return rate, additional questionnaires were sent to schools in 
September, with a letter thanking those who had contributed and 
requesting completion and return of questionnaires by those schools 
that had not participated earlier in the year. 

The questionnaire on the new Year 7 Mathematics test was sent to 
66 urban primary schools in November 1988. In January 1989 a copy 
of this questionnaire was sent to schools that had not returned the 
questionnaire and a note of appreciation sent to those schools that 
had participated. 

Interviewees included teachers, principals and assistant principals, 
in schools and a number of Departmental personnel from all regions 
of the Territory. Visits were made to a total of 19 urban primary 
schools in Darwin, Palmerston, Katherine, Tennant Creek, Nhulunbuy 
and Alice Springs. There are 66 urban primary schools in the 
Northern Territory i.e. 29% of primary schools were visited. Visits 
were made to 10 of the 77 Aboriginal schools located throughout the 
Territory. (13% of Aboriginal schools were visited). The interviews 
were conducted April to October 1988. Discussions were held with a 
total of 15 Departmental personnel (other than school staff) and 
some of the people who had worked on the development of materials. 

2.2.6 Response Rates 

Completed questionnaires were received from: 

38 principals/senior staff in urbari schools 
(representing 57% of the 66 schools); 

52 teachers of Year 5, 6 and 7 in urban schools; 

52 staff in Aboriginal schools (37% of the 77 schools); 
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43 schools involved with the 1988 Year 7 Mathematics 
Test (65% of the 66 schools). 

Note: It was difficult to determine the response rate for 
questionnaires from teachers of Years 5, 6, and 7 as many teachers 
reported teaching multi-grade classes and in one case a 
•questionnaire had been completed at a meeting of teachers of Years 
5, 6 and 7 and this was forwarded as a group response to the 
questionnaire. Information from this questionnaire was not included 
in the analysis. 



2.3 Data Presentation and Analysis 

Chapter 3 is principally concerned with presenting information 
obtained from literatures searches, Departmental records and 
program documentation. 

The information presented in Chapter 4 comes mainly from the 
questionnaires and from interviews with principals, assistant 
principals and teachers in schools. Additional information was 
obtained through visits to schools and in discussions with 
Departmental personnel. It must be noted that the information 
presented is based mainly on the perceptions of school staff on the 
program. 



3.0 AN OVERVIEW OF THE PROGRAM AND ITS OPERATION 



3.1 Aims of the Program 



Since the first introduction of the Primary Assessment Program, the 
aims have subsequently been Vevised or reinterpreted in various 



The original aims as defined by the working party set up in 1982 
were: 

(i) To assess student achievement in the essential s/c///s 
and understandings identified in the English and 
Mathematics core curricula. 

(ii) To identify broad areas of strength and weakness in 
student performance in the core components in order 
that remedial provisions for student deficiencies can 
be made at both the school and system levels. 

(Hi) To provide guidance and assistance to teachers in the 
assessment of student performance in the total 
curriculum. 

(iv) To provide a basis for curriculum development and 
evaluation (Working Party 1983). 

Additional statements on the aims include: 

From the beginning the main thrust of this program has 
been to assist and improve instruction in the core 
curriculum in reading and mathematics. A second 
important feature of this program is the monitoring of 
standards of achievement in the basic skills of reading 
and mathematics (Wee, 1986 Foreword). 

Essentially the program had two aims: to provide a 
diagnostic tool for teachers, schools and the system so 
that strengths could be capitalized upon and attention 
given to areas of weakness, and to allow overall 
assessment of progress (Payne, 1988). 

It appears that the two main aims of the program that have evolved 
are: - 



documents relating to the program. 
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the provision of quality assessment materials for 
teachers in the upper primary school to use; and 



the monitoring of student performance in 
Mathematicr and English at the upper primary 
school level. 

This study considers t..e achievement of these two aims, and in 
particular the views of school staff in regard to them. 



3.2 Developing the Item Pools 

The testing program was aimed at assessing student performance on 
the IVIathematics and English core curricula. To this end, the 
Mathematics and English item , pools were . developed. These 
consisted of criterion-referenced tests covering the core objectives 
for Year 5 and Year 7 levels. Tests were developed by panels 
involving teachers, Departmental officers and two teacfiers 
seconded full time to the then Educational Services Branch for the 
project. Wori< began on the item pools for urban schools in April 
1983. 

For the original item pools the multiple choice format was chosen. 
Reasons for this choice were: 

(i) the tests are easy to mark; 

(ii) the use of plausible alternatives makes the result 
amenable to diagnosis, i.e. the alternatives are 
generated from common errors made by students; 

(Hi) it is relatively free from response sets, i.e. students 
generally do not have a tendency to favour a 
particular alternative when they do not know the 
answer. 

(iv) it is both efficient and economical, i.e. a number of 
areas can be tested within a short time (Wee, 1985, 
p. 10). 

The items were field trialled and results analysed to provide 
information on measurement characteristics such as the difficulty 
levels, discrimination indices and reliability (Wee, 1985, p. 13). 

From 1984 to 1987 all urban primary scfiools were instructed to use 



selected tests from the item pool, photocopy the required number 
and administer these at the end of each year. A sampling matrix was 
employed for collection of test data, i.e. rotating sets of tests that 
each school uses. The same process occurred with Aboriginal 
schools in 1986 and 1987. This process changed in 1988 whereby 
originals of the tests were sent to schools, " rather "than schools 
selecting from the item pools. Some of these tests were taken from 
the item pools, in addition 'to new t'^sts being developed. The exact 
changes to the testing program are explained in 3.5. 



3.3 Materials and Support Documents 

The Primary Assessment Program materials comprise a package 
covering three assessment approaches, namely Type A: item pools; 
Type B: moderation of student's writing; and Type C: strategies for 
assessing reading. 

The itfim pools are collections of criterion-referenced items which 
have been trialled and calibrated so that their measurement 
properties are known. 

The moderatinn booklets contain annotated samples of children's 
writings which have been judged on a criterion-referenced 
framework. This framework which includes the criteria for judging 
student's writing is described in the booklets. 

"Strategies for Assessing Reading " contains assessment devices and 
suggested activities e.g. observation schedules, checklists and 
practical tasks, which teachers can use to evaluate student 
performance on tasks not assessed by pen and paper tests. 

Ten major documents have been developed so far for the program: 

1 NT Children's Writing Years 5 and 7, Moderated Samples, 1983. 

2 Children's Writing in the Northern Territory, Moderated 
Examples of Reports and Narratives from Students in Years 5 
and 7, 1988. 

3 Phase 2 Northern Territory Primary Mathematics Item Pool 
1983. 

4 Phase 3 Northern Territory Primary Mathematics Item Pool, 
.1983. . 



5 Stage 5 Northern. Territory Primary Reading Item Pool, 1983. 



6 Stage 7 Nortliern Territory Primary Reading Item Pool, 1983. 

7 Type C. - Some Strategies for Reading Assessment, 1986. 

8 Nortliern Territory Prim.ary Assessment Program for .Aboriginal 
Scliools. Mathematics Item ,^ools, Phase 2, 1986. 

9 Stage 5 Reading Item Pool, (for Aboriginal schools) 1986. 

1 0 Aboriginal Children's Writing Years 5 and 7, Moderated Samples, 
1985. 

The booklet "Children's Writing in the Northern Territory. Moderated 
Examples of Reports and Narratives from Children in Aboriginal 
Schools" (1988), is in the process of being printed and will be sent, 
to schools in 1989. 

Each document is self contained with guidelines and instructions for 
administration. They also contain introductory information 
describing the assessment program, its development and reasons for 
its use. Documents 1 - 7 have been sent to urban primary schools 
and documents 7-10 have been sent to Aboriginal schools. (Note: 
"Type C Strategies for Reading Assessment" was sent to both 
Aboriginal and urban schools). 

In addition to the above materials a set of approximately 250 
Mathematics test items, classified into the areas of space, 
measurement, number, number relations, and graphs and statistics, 
were issued to schools at the end of 1988. These test items were 
similar to those developed for the Year 7 Mathematics test 
administered in 1988. 

The following is a brief description of the contents of each of the 
documents. 

1 NT Children's Writing Years 5 and 7. Modera tRd Samples. 1983 

This booklet contains a collection of moderated examples of 
students' writing from urban schools in the Territory. A framework 
for assessing written expression is provided along with a checklist 
for teachers to use. 

Development of this assessment package involved obtaining teacher- 



rated examples of student writing on three set topics. Using the 
same rating framework as used by the teachers the samples were 
marked by a moderating committee. As teacher ratings and the 
ratings of the moderating committee were consistent in gradings, 
this, can be. considered to be a reliatDle instrument for assessment. 

2" Children's Writing in the Northern Territory-Moderated Examples 
of Reports and Narratives from Students in Years 5 and 7. 1988 

This booklet contains a selection of moderated sam.ples of report 
and narrative writing of Year 5 and Year 7 students from urban 
schools and is designed for teachers to use as a guide for assessing 
student writing. Frameworks and checklists for assessing factual 
report writing and narrative writing, along with descriptions of the 
different genres, are provided. 



3 Phase 2 Northern Territory Primary Mathematics Item Pool 
1983 

Tests in this package are based on core curriculum objectives at the 
Phase 2 (Year 4/5 level) for urban schools. Most of the items are in 
the multiple choice format. Answer sheets and examples of how to 
mark the tests are included. 

Contents: 



Core Goals/Objective Item Pool Code No. of Tests 

2-D Shapes S-A 1 

Measurement M-0 3 

Length M-A 1 

Area M-B 3 

Mass M-C 1 

Time M-D 8 

Addition and Subtraction (Written) N-C 4 

Multiplication and Division (Mental) N-D 4 

Multiplication and Division (Written) N-E 4 

Relations and Number Systems N-E 1 2 

Chance and Statistics N-F 2 



4 Phase 3 Northern Territory Primary Mathematics Item Pool. 



Tests in this package are based on the Phase 3 core curriculum (Year 
6/7" level) for urban primary schools. Most of the- items- are- -- 
presented in the multiple-choice format. Answer sheet and example 
of marked tests are included. - - 



Contents: 



Core Goals/Objective Item 


Pool Code 


No. of Tests 


3-D Shapes 


S-A 


1 


2-D Shaoes 


S-B 


8 


Position and Movement 


S-C 


5 


Paths 


S-D 


2 


Measurement 


M-0 


. . . 4 


Length 


M-A 


7 


Area 


M-B 


2 


Volume and Capacity 


M-C 


2 


Mass 


M-D 


4 


Time 


M-E 


6 


Addition and Subtraction (Mental) 


N-A 


20 


Addition and Subtraction {Written) 


N-B 


1 1 


Multiplication and Division (Written) 


N-D 


8 


Relations and Number System 


N-E 


1 3 


Chance and Statistics 


N-F 


7 



5 Stage 5 Northern Territory Primary Reading Item Pool. 1983 

These assessment materials which include; multiple choice 
comprehension tests, dictionary skills items, exercises in reading 
recipes, maps, TV programs, advertisements etc. are based on the 
Stage 5 English curriculum for urban schools. For each test, there is 
a set of materials for the teacher and for the student. Answer keys 
and student profile sheets are included for record keeping. 



Contents: 

Comprehension passages and questions C1-C7 

dictionary Skills D1-D2 

Reading for Different Purposes R1-R13 




6 Sta ge 7 Northern Territory Primary Reading Item PoqI.1983 



These assessment materials which include; multiple choice 
comprehension tests, dictionary skills items, exercises in reading 
recipes, maps, TV programs, adyertisements etc., and cloze 
exercises," are based on the Stage 7 English 'curriculum. Examples of 
test marking and interpretation of test scores in terms of mastery 
are included. " ■ ■ " 

Contents: 

Comprehension passages 01-09 

Dictionary Skills D1-D4 

Reading for Different Purposes R1-R11 



7 Type C Some Strategies for Re ading Assessment 1986 

Each Type '0' objective or group of related objectiyes for Stage 5 and 
Stage 7 is listed with accompanying practical tasks and activities 
for students. Where applicable, the appropriate strategies for 
teachers to use in administering these activities are included. 

Contents: 

The first section of this document contains 29 tasks and activities 
based on the Year 5 and 7 reading core objectives. The suggested 
activities teach and assess mastery levels in areas such as; reading 
for meaning, reading over and increasing time span, using a 
dictionary, using a Junior Thesaurus, using and understanding a 
library. 

The second section, which is based on Stage 7 only, provides 
materials and suggested activities for teaching and assessing such 
skills as summarising a piece of work, recognising an authors' 
viewpoint etc. There are 10 units in this section. 



8 Northern Territory Primary Assessment Program for Aboriginal 
S chools. Mathematics Item P ool. Phase 2. 1986 

Tests in this package are based on core curriculum objectives at the 
Phase 2 level for Aboriginal schools. The package contains tests of 
core content and skills. The tests are cross-referenced where they 
measure the same objective. Most of the test items are in the 
multiple choice format. Answer sheets and examples of how to mark 
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the tests are included, along with student profile sheets for 
recording progress. 



Contents: 



Core Goals/Objective 



Item Pool Code 



No. of Tests 



2-D Shapes 

Measurement 

Length 

Area 

Time 

Subtraction (Mental) 
Addition (Written) 
Multiplication (Mental) 
Multiplication (Written) 
Relations and Number System 
Chance and Statistics 



S-A 
M-0 
M-A 
M-B 
M-C 
N-A 
N-B 
N-C 
N-D 
N-E 
N-F 



3 
8 
7 
4 
4 
5 
1 0 
2 



3 
3 



9 Northern Territory Primary Assessment Program for Aboriginal 
Schools Stage 5 Reading Item Pool 

These assessment materials which include; multiple choice 
comprehension tests, dictionary skills items, exercises in reading 
recipes, maps, indexes, adyertisements etc., and cloze exercises, are 
based on the Stage 5 English core curriculum for Aboriginal schools. 
In the deyelopment of materials effort was made to reduce culture- 
bias. Answer keys and student profile sheets are included for record 
keeping. 

Revision of the comprehension passages and questions was done in 
1988. The revised materials were sent to schools for 
administration in the annual testing program. 

Contents: 

Comprehension passages and questions C1-C11 

Dictionary Skills D1-D3 

Reading for Different Purposes R1-R16 



Cloze 



CL1-CL8 




1 0 NT Aboriginal Children 's Writing Staoe 5. Moderated English 
Samples 1985 



This booklet contains moderated samples of Aboriginal children's 
writing and is designed for teachers to use as a guide for teaching 
and assessing writing. The samples have been categorized into low, 
medium and high competency at the Stage 5 level for Aboriginal 
schools. 

A framework for assessing written expression at the Stage 5 level 
and a checklist for teacher use are included. 

Children's Writing in the Northern Territorv. Moderat ed Examples of 
Reports and Narratives from Children in Aboriginal Schools. 1989 fin 
prgSS) 

This booklet contains a selection of moderated samples of report 
and narrative writing of Aboriginal children from Year 5 to post 
primary and is designed for teachers to use as a guide for assessing 
student writing. Frameworks and checklists for assessing factual 
report writing and narrative writing, along with descriptions of the 
different genres, are provided. 

All of these documents have been sent to appropriate schoois, along 
with instructions for their use. Support documents have included 
letters and circulars from the Curriculum and Assessment Branch 
and the Board of Studies. 



3.4 Introducing the Program to Schools 

In March 1983 every Darwin primary school principal was invited to 
an inservice where the objectives and methodology of the Primary 
Assessment Program were explained. This was followed up by a half 
day seminar involving principals, assistant principals, a 
representative from the Northern Territory Teachers Federation 
(NTTF), and all Darwin based regional superintendents. Similar 
seminars were held in Katherine, Tennant Creek and Alice Springs. 

There was regular consultation with groups such as the Counci' o* 
Government Schools Organization (COGSO), NTTF, The Primary 
Principals Association, and the Mathematics and English Subject 
Areas Committees. Such bodies received monthly statements on the 
developments of the program. 

In March 1984 inservice workshops were held for principals and 
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assistant principals of all urban primary schools. In September 
1984 urban primary schools were directed to administer specified 
tests from the item pools which were now in place in the schools 
and send the results to the Department for analysis. In 1984 every 
primary school was to administer all of the specified tests. -All but 
three schools took part in the exercise. Those schools not taking 
part were directed to submit an alternative testing program for 
their Year 5 and Year 7 classes. Only one school made a submission 
for an alternative program and this was rejected by the Department 
on various grounds but chiefly because the Mathematics program was 
not relevant to the NT core objectives for Years 5 and 7. 

At the beginning of 1986 materials were sent to Aboriginal schools. 
Comprehensive descriptions and instructions for the use of these 
materials were included, but there was no inservice as happened 
with the urban program. Some teachers had a knowledge of the 
materials as they had been on item writing panels or had been 
involved with trialling the materials. 

In the ensuing years, additional assessment materials were 
developed and sent to schools. These included: the Type C package 
"Some Strategies for Reading Assessment" (1986 for Aboriginal and 
urban schools); and "Children's Writing in the Northern Territory - 
Moderated Examples of Reports and Narratives from Students in 
Years 5 and T (1988). 



3,5 Implementing and Reporting on the Program: 
1984 CO 1989 

The testing program was designed to monitor academic standards of 
Year 5 and Year 7 students in urban schools and Year 5 to post- 
primary students in Aboriginal schools. Tests have been 
administered annually with student answer sheets being returned to 
the Department for marking, analysis and reporting. 

1984 Urban Schools. All urban primary schools were directed to 
administer a selected battery of tests to students in Year 5 and 
Year 7. Year 5 students could be excluded from the tests if the 
schooi deemed them to be working below the core curricula levels of 
Stage 5 in English and Phase 2 in Mathematics. All Year 7 students, 
defined as "children who are judged by the school to be in their final 
year of primary school" were to sit the tests (Wee, 1985, p. 15). 

The 1984 testing program included; 48 Phase 3 (Year 7 ) 
Mathematics tests, 36 Phase 2 (Year 5) Mathematics tests, 27 Stage 



7 English reading tests and 22 Stage 5 English reading tests . 

A method of stratified sampling was employed for test 
administration whereby schools were assigned to four groups, each 
group comprising large, medium and small schools b?ised on total 
- enrolment. Tests were randomly selected -from the ite.n pools and. 
then randomly assigned to each of the four groups of schools. 
Directions were sent to schools listing the tests that they were to 
take from the item pools, photocopy and administer. "No standard 
administration procedures were provided to teachers as the 
administration of these tests was seen as part of the normal 
instructional process. No time limits were specified: teachers were 
asked to use their own judgement in deciding whether enough time 
had been given" (Wee, 1985, p. 19). When all tests were 
administered schools were required to return students' answer 
sheets to the Department. Information from each school was kept 
confidential by a system of coding with only the project co- 
ordinator having access to school codes and student numbers. 

Forty nine urban schools were identified for participation in the 
program, excluding Alice Springs and Katherine Schools of the Air. 
Three schools did not take part in the testing program in 1984. 

The first ' Report on the Performance of students in the Assessment 
Program for Urban Primary Sch ools in the Northern Territory: 1984'. 
was published in 1985. This comprehensive report gave full 
descriptions of criterion-referenced testing, the development of the 
item pools, how the data was collected and a detailed analysis of 
results. 

Results were reported for each test in the form of the percentage of 
Territory students at each year obtaining 60%, 70%, 80%, 90% and 
100% correct. A copy of the major report and summary report was 
sent to all primary schools. (See Appendix VIII 1984 Results. Wee. 
1985, pp. 77 - 83, 101 - 104, .118 - 120). 

General comments on the outcomes of the tests were that both Year 
5 and Year 7 students were achieving reasonably well, on core 
objectives, in reading comprehension, did poorly in dictionary skills 
and performed satisfactorily in reading for different purposes (Wee, 
1985, pp. 116 - 117). 

In Mathematics at the Phase 3 level students performed well at: the 
four operations involving whole numbers; addition and subtraction of 
fractions with like denominators; selecting correct units/symbols 
for mass, length and volume; relations between 1 mL and 1 L, 1 g and 
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1 kg; and solving word problems involving the four operations. 
Students performed poorly in the areas of: measurement of angles; 
unit conversions in length and mass; finding intervals between two 
given times; identifying place values; and interpreting bar and line 
. . graphs (Wee,. 1985, pp. 26, . 27). _ - - - 

At the Phase 2 level in Mathematics Year 5 students performed well 
in; the four operations involving whole numbers and decimals, 
reading a calends..', identifying place values, showing order relations 
between numbers, and reading bar graphs. They did not do well in; 
comparing areas, reading time on a clockface, and rounding numbers 
(Wee, 1985, pp. 61, 62). 

A summary report was produced from the 1984 data. This report 
contains abbreviated information on the results and general 
comments on the performance at the two year levels on the sets of 
tests. The report indicates that the performance of students on the 
core objectives in Mathematics and English reading was generally 
satisfactory. The results indicate that most students were 
competent in most of the core objectives (Wee, 1985 ). 

1985 Urban Schools. The method of matrix sampling and rotating 
the tests between the four groups (as determined in 1984) was 
employed. The same tests as used in the 1984 program were to be 
used and a list was sent to each school detailing the tests they were 
to administer for that year. Fifty one schools were identified for 
inclusion in the testing program. 

The ' Report on Student Performance in the Northern Territory 
Primary Assessment Program for Urban Schools: 1985 ' was written 
in 1986 when the results had been analysed. As with the 1984 
report, tables of results at the different cutoff levels were 
included, along with discussion of performance on selected 
objectives. An additional feature was the comparison of the 1984 
and 1985 results. 

Results indicated that in 1985 the majority of students in Years 5 
and 7 were competent in most areas of the core curriculum in 
Mathematics. "Most students in Year 7 did not appear to have a 
problem in the four operations and could apply thesB in solving very' 
simple word problems. But quite a high proportion seem to have 
difficulty in relations such as the conversion of units in which 
decimals are involved." Most of the year 5 students were successful 
in most areas of the core with the exception of some aspects of 
subtraction and division. "Analysis of the results :n Reading show 
that most students in both Years 5 and 7 were competent in the 
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skills required to locate information from a telephone directory, 
follow directions on recipes, interpret information from various 
sources such as a street map, a concert program, TV program or use 
library reference skills However, in the areas of comprehension of 
selected passages and dictionary skills, performance was somewhat 
.^poorer" (Wee,: 19.86, p. 10.3). . 

Comparison of the 1984 and 1985 results indicated improvement in 
some Mathematics areas at both the Year'5 and Year 7 level. Results 
in the reading comprehension involving selected passages showed 
little change, with results in the dictionary skills tests being poor 
and showing little change from 1984. Results on tests involving 
reading for different purposes showed improvement at the Year 7 
level and decline for Year 5 (Wee, 1986, pp. 103, 104). The 1985 
summary report described the areas of strength and weakness, 
percentages achieving competence, and comparison between the 
1984 and 1985 results. 

1986 Urban Schools. The number of tests to be administered by 
urban schools was reduced significantly. The reason for this change 
was due to feedback from schools which indicated the testing 
program was a "heavy burden on resources in terms of time and cost 
of photocopying test material" (Wee, 1987, p. 28). The 'Report en 
Student Performance in the Northern Territory Primary A ssessment 
Program for Urban Srihnols: 1986' describes student results on 28 
Phase 3 Mathematics tests, 33 Phase 2 tests, 20 Stage 7 English 
reading tests, and 17 Stage 5 reading tests. Tests for the 1986 
program were selected on the basis of their value to students for 
future learning and whether they had sound statistical 
characteristics as indicated from previous years of administration. 
As with previous years, results showed areas of strength and 
weakness in the two subjects. 

Comparisons were made between 1985 and 1986 on those tests that 
were common to both years. There was little evidence to indicate 
improvement or decline in performance (Wee, 1987). 

1986 Aboriginal Schools. 1986 was the first year of the testing 
program for Aboriginal schools. The assessment materials for 
Aboriginal schools i.e. the English and Mathematics item pools, the 
Type C docL'ment and the first writing moderation booklet for 
Aboriginal schools, were sent at the beginning of 1986. All 
Aboriginal schools including Yirara and Kormilda colleges were 
included in the program. Schools were assigned to two groups using 
random stratified sampling. A total of 17 Mathematics tests and 10 
reading tests were randomly assigned to each group of schools i.e. 
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each school had to administer 8 or 9 Mathematics tests and 5 
reading tests. In September a letter was sent to schools directing 
them to select, photocopy and administer specified tests from the 
item pools and return students' answer sheets to the Department. 
The same set of tests was to be administered to students from Year 
5 to post-primary who the school considered were working at the 
Stage 5 level in English and Phase 2 in Mathematics as specified in 
core curricula for Aboriginal schools. "A set of guidelines was 
provided to teachers to ensure some uniformity in test 
administration. ...Teachers explained how students should write their 
answers for multiple-choice and open-ended items. Students were 
not allowed to talk with each other during the test and teachers 
were not to help students answer any test questions." No time limit 
was specified (Huang, 1987, p. 12). 

At least thirty schools did not administer the tests. It appears this 
was due to union bans and the fact that schools felt they did not 
have students working at the specified levels necessary for 
attempting the tests. "From figures supplied by schools that sent in 
their returns, about 400 students from Year 5 to post-primary were 
not tested because they were not working at the level of the core 
and in their teachers' opinion would not have been able to cope with 
the tests (Huang, 1987, p. 17). 

Results indicated serious weaknesses in both Mathematics and 
English reading. In the majority of the tests 60% of students did not 
reach the competence scores of 70% in English and 80% in 
Mathematics. (See Appendix XI for summaries of results) 

Reports were distributed to schools and Departmental personnel 
with a request from the Secretary of Education for action for 
improvement: 

Although many students did not take the tests, the report 
must create a great deal of concern regarding the poor 
achievement levels shown by ■ students in Aboriginal 
schools. It is obvious that there is an urgent need to 
develop strategies for overcoming weaknesses identified 
in the report. All schools, head office staff and the 
various interest groups which receive the report need to 
address the problems that have been revealed (Huang, 
1987, Foreword). 

1987 Aboriginal Schools. The same 17 Mathematics tests and 10 
English tests were rotated between the two groups of Aboriginal 
schools and administered to students from Year 5 to post-primary at 
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the end of the year. "Less than a quarter of the target population of 
around 2 500 students took the tests. Only a third of the total 
number of schools sent in their students' answer sheets for data 
analysis" (Huang, 1988, p. 13). 

The reason for the low participation rate is believed to be bans on 
the testing program by the Northern Territory Teachers Federation 
and the fact that many students were not operating at the required 
'core' levels. 

Results in both Mathematics and English were very poor, showing a 
decline in performance from the previous year (Huang, 1987, p. 13). 

1987 Urban Schools. The number of tests were slightly reduced 
for the urban testing program and it was decided to eliminate the 
dictionary skills tests as it was felt that these skills were better 
tested at the classroom level in a more practical way. Twenty seven 
Phase 3 Mathematics tests, 34 Phase 2 Mathematics tests, 16 Stage 
7 reading tests, and 15 Stage 5 reading tests were randomly 
assigned to the four groups of schools i.e. each school administered 
approximately one quarter of the number of tests. Fifty three 
schools were identified for inclusion in the program and were 
instructed to select and photocopy the tests and administer them 
during the last semester. Most of the urban schools took part in the 
program. 

The results were satisfactory, with most children achieving 
mastery in both the Mathematics and English tests. Comparison with 
previous year's results was made oh common tests from the previous 
administrations. Because the areas for comparison were limited to 
some topics in Mathematic and to specific areas in reading i.e. 
reading comprehension of passages and reading for different 
purposes, readers were cautioned against making statements about 
decline or improvement in overall standards in these subjects 
(Huang, 1988, pp. 23 -27, 30 -33). 

1987 Urban and Aboriginal Schools. For the first time, each 
school (Aboriginal and urban) received a computer printout of its 
individual results, along with the Territory results for 1987. These 
were sent to schools in March 1988. Individual schools received 
only their own school's results. The project co-ordinator is the only 
person with knowledge of all school results as there is strict 
security to keep these confidential. 

1988 Urban Schools. In 1988 there were some major changes to 
the testing program for u'ban schools. The Phase 3 Mathematics 
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tests were replaced with a 60-item Mathematics test based on the 
Board Approved Curriculum for Year 7. Multiple copies were sent to 
schools for administration on the 19th October 1988 with a time 
limit of 75 minutes. All of the English comprehension tests were 
revised. Schools were required to administer only two reading 
comprehension tests, one reading for different purposes test and a 
commercial TORCH (Test of Reading Comprehension) test. The 
sampling matrix was employed for administration of the Stage 7 
English tests. Selected originals of the test material were sent to 
schools for photocopying. Multiple copies of the TORCH test were 
sent to all schools for administration. Schools were asked to mark 
the TORCH test and return the students' answer sheets to the 
Department i.e. for this test only the teachers' marking was 
accepted as correct and. results were fed into the computer for 
analysis. Marking of this test within the Department was thought to 
be too time consuming. From the limited number of visits to schools 
during or after administration of this test, discussions with 
teachers indicated that they thought this was a good test of reading 
comprehension. 

The testing program for Year 5 urban schools was similar to the 

1987 program, however the selected originals were sent to schools 
for them to photocopy. This eliminated the need for schools to find 
the selected tests in the item pools. 

1988 Aboriginal Schools, The program for Aboriginal schools 
was also altered in 1988. All of the comprehension tests were 
completely revised and selected originals sent to schools for them 
to photocopy. A 85-item Mathematics test was constructed from 
the Phase 2 Mathematics item pool for Aboriginal schools and 
multiple copies were sent to schools for administration. Again this 
eliminated the need for schools to select the tests from the item 
pools and reduced the amount of photocopying that needed to be done 
by schools. 

Results were analysed and sent to urban schools in March and 
Aboriginal schools in April 1989. 

1989. The proposed 1989 program will include fewer tests than in 
previous years. The urban program will include: for Year 5, one 
reading comprehension test and one TORCH and one reading for 
different purposes test and one summative Mathematics test. For 
Year 7 there will be one reading comprehension test, one reading for 
different purposes test, one TORCH test and a new summative 
Mathematics test. 



The Aboriginal program will include one reading comprehension test, 
one reading for different .purposes test and a summative 
Mathematics test based on the core curriculum. 



3.6 Feedback and Evolution 

The program has undergone continual change and development. This- 
has been due to ongoing review by the Board of Studies through its 
Primary Assessment Committee which meets at least four times 
each year. The direction, of the changes has also been influenced by 
indirect input from schools through subject advisers, regional 
superintendents and such bodies as the NT Principals Association. 

Because the program is open to review, this study itself has already 
influenced changes in the operation of the program during 1988 and 
1989 e.g. the call for inservice was partially attended to by the 
running of inservice courses in Alice Springs and Darwin in 1989. In 
addition to this, changes to the 1989 testing program were 
influenced to an extent by findings of this study. 

It must be noted that some of the comments by teachers on subjects 
such as the Phase 3 urban Mathematics item pool being too simple 
for Year 7 have already been covered by the introduction of a Year 7 
test based on the Board Approved Curriculum. This evaluation was 
extended to incorporate some of these changes e.g. the survey of 
teacher opinion on the 1988 Year 7 Mathematics test. 



4.0 FINDINGS OF THE STUDY 



4.1 Introduction 

The major findings of the study, obtained from the questionnaire 
survey, the interviews and the school visits are presented in this 
chapter. In the main, the findings represent the responses and views 
of practising teacher*? and hence provide a school-based perspective 
upon the program, its operations and its associated materials. Such 
a perspective is clearly of critical importance to decision makers 
responsible for the future of the program. Neverless, in interpreting 
any of the findings, for example those relating to the achievement of 
objectives, it: is clearly necessary to recognise that teachers form 
only one stakeholder group and that the views of others might well 
be different. ' It must also be recognised that the perceptions 
reported here may well have been influenced by the philosophical and 
idealogical reservations concerning system-wide testing which have 
been expressed at various times by representatives bf teacher 
organisations. 

Data from school staff are presented in three main categories: 

(a) Senior staff from urban primary schools (US) 

(b) Teachers in urban primary schools (UT) 

(c) Staff in Aboriginal Schools (AS) 

(Note: It was decided to combine the information from seniors in 
Aboriginal schools with that of teachers in Aboriginal schools as 
comments indicated that seniors had a high teaching !oa^, and were 
making comments as teachers and vice versa. For questions that 
seniors only were asked to answer an additional category for seniors 
in Aboriginal schools (SA) was used and for questions which 
teachers only in Aboriginal schools were required to answer the 
abbreviation (TA) was used. 



4.2 Awareness and U^e of Materials by School Staff 

The first question participants were asked was whether they had 
seen the materials. For ease of recognition, a picture of each 

25 



o 

ERIC 



35 



document was shown to interviewees and included in the 
questionnaire. The next question participants were asked was 
whether they had used the materials and for what purposes. The 
purposes listed in the questionnaire for participants to choose from 
were: 

A - system-wide tests selected from the item pools; 

B - classrooms tests and exercises selected from the item pools; 

C - assessing student writing; 

D - assessing other skills. 

Tables 1 - 3 indicate participant awareness and use of materials and 
the purposes for which the materials were used. 

Graphs 1 - 3 show the percentages of the three groups; senior staff 
in urban schools (US), teachers in urban schools (UT) and staff in 
Aboriginal schools (AS), who had seen each of the materials and had 
used these either in the past or during 1988. . . : . 

In urban schools a large majority of senior staff (approximately 
85%) had seen the writing moderation booklets and the item pools. 
Seventy one percent of seniors reported having seen the 'Type C 
document. Graph 1 shows dramatically that although a large 
majority of seniors had seen the materials, the proportion having 
used the materials, was much less, e.g. with the Type C document, 
11% of the respondents reported having used it in 1988 and 47% said 
they had used it in the . past. 

Principals and senior teachers were asked to comment on how the 
materials were used and of what value they were to the school. Of 
the 38 questionnaires received from seniors in urban schools, 18 
(47%) reported that the materials were used only for the annual 
system-wide tests with such comments as: 

Used at Year 5 and 7 as requested. 

Utilized only for set system-wide tests. 

Little of the materials have any relevance to what is done 
in classrooms. Year 5 and 7 teachers do the annual required 
tests and that's all. 



Table 1 Urban Seniors' Awareness and Use of Materials 



MATERIALS 



SEEN 



USED 1988 



USED PREV 



USED FOR 



897o 
(34) 



89% 
(34) 




trP/>f fCCf' 



7. roidJir 1396. 



NOTE 

Tha figures in brackets show 
tha numbar of positiva responses 
in each category. 



38% 

(14) 



47% 
(13) 



24% 

O) 



24% 

(9) 



16% 

(6) 



13% 
(5) 



11% 
(4) 



61% 

(23) 



37% 
(14) 



61% 
(23) 



55% 

(21) 



66% 
(25) 



58% 
(22) 



42% 

(16) 



A 
8 
0 
D 
N 



A 
B 
0 
D 
N 



A 
B 
0 
D 
N 



Total Number of Respondants 38 



A - system-wida tests selected from tha item pools 
B - classroom tests and exercises 
C - assessing student writing 
D • assessing other skills 
N - not used 
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Graph 1 Urban Seniors' Awareness and Use of Materials 





m 
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1983 



WMod 
1983 



Maths 
Phase 2 



Maths 
Phase 3 



Reading 
Stage 5 



Reading 
Stage 7 



□ 


Seen? 




Used 1988? 


M 


Used Prev? 



m 



Type C 
Reading 



Given as directed by program once a year. Not used at any 
other time and not used for diagnostic purposes when given. 

Twelve of the 38 seniors said they made other use of the materials 
with' such comments as: 



Often used as end of term assessment materiais. 



Mainly used at the end of each term to assess progress of 
concepts covered earlier. 

I have used the materials so that kids know what sort of 
tests they are going to have to do. With Maths they are not 
used to multiple choice questions. 

Having only been 18 months at this particular school, my 
efforts have been directed to making teachers aware of 
the value of the material and encouraging the use of it. 

Used by individual teachers to find weaknesses to teach to 
and provide guidelines for future programming and policy 
reviews. 
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Type C (Strategies for Assessing Reading) is useful for 
teachers. Type A and B (ttie item pools) are rarely used as 
diagnostic tools. 

The only real document of value for me is the current 
writing, moderation booklet. I use this at least 6 times a 
year i.e. the frameworks are filled in for individual 
students narratives. 

The remaining 26 seniors either said nothing or said they did net use 
the materials or they reported the materials were of little value to 
the school. Comments regarding this were: 

Early use. ..identified several errors in answer sheets and 
some poor presentation. 

System for establishing standard appeared questionable. 
Massive amount of work for questionable result. 

As required by the Department. Is of little if any value to 
the school. 

Of no benefit to the school. 

Not useful at ail for ongoing assessment for teaching 
purposes. 

During . interviews with principals and senior teachers, similar 
comments were made about the use of materials. It appeared that 
where seniors were familiar with the program, the materials were 
being used to varying degrees in that particular school. In schools 
where seniors had little knowledge of the program and thought it 
only involved testing, the materials were not being used and in most 
cases seniors had no idea what materials should be in the school or 
where these were located. Examples of comments relating to this 
were: 

The materials are recommended to staff to use. The 
writing moderation booklets are very good. 

Don't see the need for item pools, teachers construct their 
own tests. 
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Table 2 Urban Teachers' Awareness and Use of Materials 



MATERIALS 



SEEN 



USED 1988 



USED PREV 



USED FOR 



SaoKii': '.9 83. 
Suii cavar. 



62% 
(35) 



2. 3aokl«c 1981.1 rvJ 



71% 
(37) 



60% 
(31) 




6. ?old«r i9B3 
-an C3V«r. 



■i'^k^.yif 54% 

'i-. V.-';:7 (28) 




31% 

(16)- 



46% 
(24) 



12% 

(6) 



12% 

(6) 



10% 
(5) 



10% 

(5) 



10% 
(5) 



35% 
(18) 



A 4% 

B 1 1 % 

C 42% 

D. 4% 

N 57% 



71% 
(37) 



A 
B 
C 
D 
N 



4% 
13% 
47% 

6% 
47% 



42% 
(22) 



A 30% 

B 25% 

C 0% 

D 4% 

N 60% 



42% 
(22) 



A 
B 
C 
D 
N 



36% 
28% 
0% 
6% 
51% 



33% 
(17) 



A 
B 
C 
D 
N 



33% 
(17) 



19% 
(10) 



36% 
25% 
4% 
8% 
58% 



A 
B 
C 
D 
N 



34% 
1 9% 
2% 
6% 
60% 



A 
B 

C 
D 
N 



4% 
8% 
6% 
1 5% 
75% 



Total Number of respondents 52 



The figures in brackets show 
the number of positive responses 
in each category. 



A - system-wide tests selected from the item pools 
B - classrooRi te&(s and exercises 
C • assessing student writing 
D • assessing other skills 
N - not used 
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The writing moderation booklets are very good, especially 
the narrative writing. The assessment sheets are very 
good. 

In this school, as in the last ' school I was in, the 
impression that was strongly held was that the materials 
were not to be used for anything else except testing. 

A major hurdle is the fact that the folders were required 
for the tests. Now that this is not the case, with the 
Department sending schools the tests that are needed, 
schools must be shown how they can use the documents. 



'Sell' the program to AP's (assistant principals), 
of a video is excellent. 



The idea 



Urban teachers were asked whether they had seen the various 
assessment documents and whether they had used the materials in 
1988 and in previous years. They were also asked to comment on the 
purposes for which the documents were used and whether they 
thought the documents of value to their teaching. Graph 2 and Table 
2 summarise their replies. 

Graph 2 Urban Teachers' Awareness and Use of Materials 
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Fewer classroom teachers than seniors had seen the range of 
materials. Twenty two of the 52 respondents (42%) reported that 
they had seen the Type C document, approximately 60% had seen the 
reading and Mathematics item pools and 72% had seen the 1988 
writing moderation booklet. However, a. much, smaller percentage of 
the respondents reported that they had used the materials, either in 
the past or during 1988. Approximately 35% of the respondents had 
used the item pools in past years with only about 10% having used 
them during 1988. Graph 2 demonstrates this dramatically. The 
exception to this pattern is with the writing moderation booklets. 
As the graph indicates over 70% of urban teachers had seen the 
1988 booklet and 46% had used it. Half of the 60% who had seen the 
1983 writing moderaiion booklet reported they had used it. 

With regard to the teachers who had used the Mathematics and 
reading items pools (see Table 2), approximately 35% reported 
using them for the system-wide testing exercise with 
approximately 25% saying they had used them for class tests or 
exercises. 

Fewer teachers reported having seen the the Type C document 
'Strategies for Reading Assessment' than any af the other 
documents. Only five respondents (10%) said they had actually used 
this document in 1988 and only 10 (20%) had used it in previous 
years. 

It is obvious from some of the selections made by teachers on the 
questionnaires as to the purposes for which materials were used, 
that they had no knowledge of the materials in question as they 
could not be used for the purpose selected. For example 4% of 
teachers indicated that the writing moderation booklets and the 
Type C document were used for system-wide testing. This is a 
basic misconception, since only the item pools were used for this 
purpose. 

During visits to schools it was very often necessary to show the 
materials to teachers and explain how they could be used as the 
teachers had little knowledge of program materials. 

The following are examples of comments made by teachers 
demonstrating their lack of awareness and use of materials: 
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The teachers aide was responsible for photocopying the 
tests so the folders were just, buried in the store. I 
didn't realize they could be used for everyday lessons, I 
wish I had. 

The fact that the senior or Assistant Principal is 
responsible for administering the tests, has meant that 
the documents have been reserved for the end of year 
tests and that teachers have not had the opportunity or 
been encouraged to use them as resource material. 

There is lack of knowledge and understanding of the 
program. It needs to be explained to 'senior management'. 

Those teachers who had used the materials had found them quite 
useful, as is indicated in the following comments: 

/ use the Maths item bank to assess core mastery prior to 
class evaluation of more difficult content Reading banks 
are used along with other class constructed evaluations. 

The hJiaths item pools can be used either to test or teach a 
concept. Also the 'breaking down' of Maths concepts into 
several sections or aspects psychologically is appealing 
as it appears to make teaching a particular concept much 
easier. On the other hand if used for testing, it helps to 
pinpoint more accurately any areas of difficulty. 

On several of the questionnaires that had not been completed, there 
were comments such as the following: 

New to the Territory, have not seen these materials. 

Didn't know they existed. 

Cannot comment as I have not used these. 

Unable to comment as these materials do not appear to be 
in this school. 



Table 3 Awareness and Use of Materials - Aboriginal Schools 



MATERIALS 



SEEN 



USED 1988 JSEDPREV USED FOR 




y«Xlcw caver 




85% 

(44) 



31% 
(16) 



79% 

(41) 



37% 

(19) 



73% 

(38) 



9% 

(5) 



42% 

(22) 



13% 

(7) 



50% 

(26) 



17% 
31% 
C 0.5% 
D 8% 
N 21% 



50% 
(26) 



A 
B 
C 
D 
N 



27% 
(14) 



12% 

(6) 



1 9% 
27% 
4% 
S% 
23% 



A 2% 

B 6% 

C 1 7% 

D 0% 

N 35% 



4% 
8% 
2% 
4% 
39% 



Total Number of respondents 5; 



The figures in brackets show 
the number of positive responses 
in each category. 



A 
B 
C 
D 
N 



system-wide tests selected from the item pools 
classroom tests and exercises 
' asi>essing student writing 
. assessing other skills 
not used 



34 



ERIC 



44 



Graph 3 Awareness and Use of Materials - Aboriginal Schools 




□Seen? 
13 Used 1988? 
Used Prev? 



Maths Ph.2 Reading St.5 Writ. Mod. Type C 
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As can be seen from the graph, a large number of staff (74% - 85%) 
had seen the item pools and the writing moderation booklet. 
However, only 43% of staff had seen the Type C document. Fifty 
percent of staff had used the item pools in the past with a smaller 
percentage having used them in 1988. Table 3 indicates that of 
those who had used the item pools approximately 18% had used 
them for the annual testing and approximately 30% had used them 
for class tests and exercises. Selected examples of comments 
about the use of materials follow: 



To test a certain procedure (e.g. addition). For end of year 
testing. Quite valuable for older children - no benefit for 
younger ones. 

They are useful as a class test after completing relevant 
work beforehand. 

The materials have been used to familiarize children with 
test materials and ascertain areas of concern and poor 
performance. 



I used it to work out the standard of the children where I 
didn't have any other children for comparison. 
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/ have used the reading and Mathematics item pools 
several times during the year as examples for the children 
before giving them the actual ones used for the 
assessment. 



Only a small percentage of staff from Aboriginal schools tiad used 
the writing moderation booklet. Nine percent had used the bool<let 
in 1988 and 27% reported they had used it in the past. Similarly, 
very few had used the Type C document. Thirteen percent said they 
used the document in 1988 and 12% said they had used it 
previously. 

Comments on several of the questionnaires and during interviews 
provided reasons for questions concerning materials not being 
answered. An example of such comments were: 

/ have not seen them. 

I am only finding out about it all just now. 

As I have just become teacher in charge I have not seen 
any of the materials. 

Have seen the materials but not used any. 

They are not used. 

So far these assessment materials have not been used. 

Very little use or value, except for some advanced primary 
students or post primary. 

Of the 10 Aboriginal schools visited, only two appeared to have the 
full set of documents. In the other eight schools at least one of the 
documents could not be located. 

NOTE: It must be noted that some respondents either did not 
understand the question concerning the use of materials or they 
lacked knowledge of the mitenals as some reported using the 
materials for purposes for which these materials are not suited, 
e.g. 13% of urban seniors, 8% of urban teachers and 4% of staff 
from Aboriginal schools reported using Type C for system-wide 
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tests. Similar numbers reported writing moderation booklets were 
used for system-wide tests. This is a misconception as these two 
sets of materials were designed for school use only. 



4.3 Materials - Their Place in Schools 

The information presented in this section of the report was 
obtained mainly through observation and interviews while visiting 
schools. As only a percentage of the schools was visited, a degree 
of caution should be observed in generalizing from this data to the 
Territory as a whole. However, from discussions with subject 
advisers and regional superintendents, and checks lor possible bias, 
there appears to be no reason to believe that the following picture 
of the place of materials in schools is atypical. 

As explained earlier in the report, the Mathematics and English 
item pools were sent to urban schools in 1984 and to Aboriginal 
schools in 1986. Because it was explained to schools that the 
annual tests were to be taken from these documents, they were 
guarded by school seniors and kept 'safe' so that tests could be 
photocopied at the end of each year. In the majority of schools 
visited, it appears that multiple copies were made of all the tests 
and these were filed in special filing cabinets, some of which were 
purchased by the Department specifically for this purpose. This 
meant that many classroom teachers did not have access to the 
materials for class tests and exercises as was planned. 

As additional materials were sent- to schools, e.g. the writing' 
moderation booklets and the Type C document, these were 
frequently distributed to subject faculties and stored in various 
places around the schools. In over half of the schools visited, it 
was necessary to physically search for the documents. In only 
three of these schools were they stored in a central place within 
the school - in the Principal's or Assistant Principal's office, the 
library and the book store. 

In most cases, before carrying out interviews, it was necessary to 
describe either to the interviewee or at a staff meeting what the 
Primary Assessment Program is. Many of those inten/iewed did not 
know that the item pools were for classroom use, did not realize 
that documents such as the writing moderation booklets were part 
of the assessment program, and had the impression that the 
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Primary Assessment Program consisted of "those tests we have to 
do at the end of the year". 

From the interviews, comments which substantiate these 
observations included: 

The materials are mainly l<ept for the one thing - testing, 
(senior teacher, urban school) 

The materials should be kept in a teacher resource area, 
probably the staffroom is a good place. (Teacher, urban 
school) 

The item pools are kept in the storeroom near the front 
office. The aide is responsible for the photocopying when 
we get notice to do the tests. (Assistant Principal, urban 
school) 

Not sure that all materials are here. We used the item 
pools last year for photocopying the tests. I am not sure 
we have the rest. (Principal, Aboriginal school) 

I have never seen these materials. I can't recall them ever 
being received in this school (Assistant Principal, Area 
School). 



4.4 Teachers' Views of the Strengths and Weaknesses 
of Materials 

Urban teachers (UT) and teachers in Aboriginal schools (TA) were 
asked to comment on what they saw as the strengths and 
weaknesses of the three types of materials - the Mathematics and 
Reading Item Pools; the writing moderation booklets and "Type C - 
Strategies for Reading Assessment", A total of 52 teachers in 
urban primary schools and 27 teachers in Aboriginal schools 
returned the questionnaires. Tables 4, 5 and 6 provide a summary of 
replies. 
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Table 4 Strengths and Weaknesses of Ite'm Pools 



Strengths 




TA 


Used as referonce matarial. Provides good ideas for teaching. 
A resource. Helpful. Accessible. Useful guide for planning. 


1 7 


1 0 


Tests basic concepts. Evaluates abilities/readiness. Diagnostic. Provides 

accocciTiAnt tHmiinH ^inf^thoc nor^nn'^ ovo^ t^r^nH vsriotv 

not H UltUUMIi AllwUiOl QjOw. wlUwU VClJ lOly. 


1 9 


5 


Standardised reference for teachers. Provides uniformity to 
education system. A valid testing system. Tests seem accurate. 


1 7 


2 


Total numfaii^ rtf eanini<»nts 


S3 




No comment on strengths of item pools. 


1 9 


1 1 


Weaknesses . 






Not related to classroom learning. Maths too simple. Lower/higher levels 
nofidad Not diaanostic 


1 8 


2 


Reading - lack of interpretive material. Reading matarial not relevant to NT 


7 


6 




A 




1 OLI O Ul 1 iO WVJ 1 iwUI 1 III iM. fVCkOlO Ul Ul 1 lO< ^AfJOIIwlVO. 1 VJU lllUl^ll kjl lUlU^wU j II IM 

Late or no feedback from tests. Mistakes in materials. 


1 5 


<^ 


Other 


1 s 


2 


Total number of comments 


so 


18 


No comment on Weaknesses 


27 


12 


Total number of respondents 


52 


27 



From the table the perceived strengths of the item pools fitted into 
three major categories; useful as reference material, provides a 
standard, and has some diagnostic value. 

The main weakness reported by urban teachers was that the 
materials were not related to classroom learning. They made 
special reference to the Mathematics item pool tests which were 
considered too easy for the year levels for which they were 



prescribed. Presumably this was because the item pools did not 
test beyond the core objectives. As the tests are extended to 
assess the full Board Approved curriculum, such criticisms should 
dissapear. Another major weakness noted by both groups, although 
not directly related to the intrinsic worth of the materials, was 
that the test administration was expensive in terms of time- and 
paper in photycopying. To avoid this problem, multiple copies of 
tests are now being printed centrally and distributed to . schools... 



Table 5 Strengths and Weaknesses of Writing Moderation 



Strengths 

Realistic look at children's writing other than own school. Allows 
comparison of standards. 


UT 


TA 


22 


2 


Good evaluation system. Gives teachers a new approach to gaining written 
written materials from children. Support materials in booklets are great! 


37 


1 


Number of comments on strengths 


75 


3 


No comment on the strengths of writing moderation materials 


1 6 


1 9 


Weaknesses 






Time consuming. Doesn't cover ail genres 


7 


0 


Narrow range of samples. Final product only. 


4 


4 


Other 


6 


0 


Numbar of commants on weaknesses 


17 


4 


No comment on weaknesses.*(18 + 19) 

*1 8 made comment on strengths only, and no comment on the weaknesses. 
(It is assumed that those who found strengths did not find weaknesses in the 
writing moderation booklets). 17 made comment on both strengths and 
weaknesses and 19 made no cc.iment at all. 


37 


19 


Total number of respondents 


52 


27 



Urban teachers supplied 75 comments on the strengths of the 
writing moderation compared to 17 comments about the 
weaknesses. The two strengths of the writing moderation bocklets 
were that it provides a standard and is good assessment support 
material. 
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Thirty nine percent of respondents from Aboriginal schools saw the 
writing moderation as being very useful and excellent support 
material. This may be an indication of the method of development 
of this document i.e. teachers were directly involved in providing 
samples for the booklet. 



Table 6 Strengths and Weaknesses of the Type C Document 



Stranaths 


UT 


T A 


Good variety of ideas and activities. Non-threatening. Useful collection of 
reading materials. Good resource. 


1 1 


3 


Other e.g. Needs to be tnserviced. 


5 


0 


Number of comments 


16 


3 


No comment on the strengths of Type C or hadn't seen- 


36 
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Weaknesses 






Need to be promoted more. More inservica. 


4 


0 


Wider achievement levels at upper and lower end of scale would be 
helpful for diagnosis. Does not relate to Aboriginal education. 


1 


2 


Number of comments 


5 


2 


No comment on weaknesses. Have not seen materials. 


47 


22 


Total number of respondents 


52 


27 



It is obvious from the information presented in the tables that 
teachers had much more to say about the item pools and the writing 
moderation booklets. It appears they are much more familiar with 
these documents than with Type C where over half of the 
respondents had not seen the document or did not make a comment 
about its strengths and weaknesses. 
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4.5 Strengths and Weaknesses of the Program: Views 
of Senior Staff- 



Urban seniors (US) and principals and seniors in Aboriginal scliools 
(SA) were asked to comment on what they saw as the strengths and 
weaknesses of the program. Table 6 summarises their comments. 



Table 7 Strengths and Weaknesses of the Program 





U5 


?A 


Strengths 






Provides a standard. Gives uniformity to the education system. Good 
monitoring tool. Identifies strengths and weaknesses. Provides benchmarks . 


2 1 




Provides valuable resource material. Assists in reporting 


16 


9 


Writing moderation valuable. 


i 


i 

I 


inerQ is a neeo to tost ine system, rrovioes sysiem monnonng. 




o 

w 


Other. 


4 


1 


Number of comments on strengths 


50 


28 


No comment on strengths of the program 




Q 

o 


Weaknesses 






Time consuming. Waste of time. Expensive. Too much photocopying 


1 1 


3 


Tests do not match classroom teaching. Not diagnostic. Nothing done with 
results. Disadvantages ESL students. Aboriginal children not at level of tests. 


8 


1 5 


KInt ciinnnrtoH in tho c/^hnr^l t Ittio WnnwIaHno hv r^lflQ^rnnm tASlchor^ nflf^S 

to be insen/iced more. Not enough contact between Department and schools. 


7 


2 


Mistakes in materials. Tests pooriy designed. Students have to juggle papers. 


S 


3 


Maths item pool too easy. 


5 


3 


Other 


9 


2 


Number of comments on weaknesses 


48 


25 


No comment on weaknesses. Have not seen materials. 


4 


7 


Total Number of respondent: 


; 38 


27 



Totals of 38 seniors in urban schools and 27 seniors in Aboriginal 
schools completed the questionnaire. They offered a total of 50 and 
28 comments respectively on the strengths and 48 and 25 comments 
respectively on the. weaknesses. 

The major strength seen by each group was that it provides a 
standard and gives uniformity , to the education system. 

The major weakness of the program was the amount of photocopying 
and time required in preparation of the tests for administration. 
Other perceived weaknesses were that the testing does not relate to 
classroom learning and the tests are not diagnostic, there are 
mistakes in materials, and there has not been enough promotion and 
inservice.on the program. 



4.6 Teachers' Attitudes Towards the Program as 
a Whole 

When participants were asked to give their views on how they felt 
about the program as a whole and to provide any other comments, 
most of the replies fitted into categories of either being in favour 
of the program or not in favour of the program. Table 8 summarises 
these comments for 'the three groups - Urban seniors (US), Urban 
teachers (UT) and staff from Aboriginal schools (AS). Additional 
comments which did not fit into these two categories and which 
covered quite different subjects have been discussed further on in 
this section of the report. 

Urban seniors gave a wide range of reasons for and against the 
program. Fourteen percent thought the program worthwhile while 
14% of urban teachers saw it favourably because it provided a 
standard and reveals student inabilities. Staff from Aboriginal 
schools favoured the program because they thought it was a good 
idea but needed more work and more materials for Aboriginal 
schools. Respondents from all three groups would like to see more 
inservice on the program. 

The main reasons given for not favouring the program were that k 
was time consuming and not cost effective. The program was also 
seen to be a cause of conflict between the Teachers' Federation and 



the Department and that it was viewed as a step toward teacher 
assessment. 



Table 8 Attitudes towards the Program as a Whole 







UT 

w 1 




Comments In favour of the program 








Yvonnwnne. - neipTui. csseniiai it MDonginai scnoois arB pver lu improve 
standards. Extend it. Provides a standard. Reveals student's abilities 
and inabilities. Happy about it Very pleased. Support the program. 


4 4 

1 1 


ft 

9 


0 


rroviaes maiDriais. rine as an assessmeni looi in scnoois. wniinQ 
moderation is of value. Reading satisfactory. Year 7 Maths test is a 
siep in ine rigni atreciion. 


o 
o 


0 


o 


Basically a good idea but needs more work, i like it but It needs a 
greater range of materials for Aboriginal schools/bilingual schools. 
Adequate for its purpose. Just okay. Reasonable. 




6 


1 3 


Would like to see more visits from CAB officers and more inservice. 


3 


6 


5 


Totals 


33 


27 


29 


uommonis noi ravouring in9 program 








Doubt that it is cost effective. Waste of time and money. 


14 


9 


3 


Nuisance. Imposition of a testing system. Creates more work and 
stress for teachers. Not a worthwhile activity. Centralized control. 
Inappropriate. My feelings towards the program are very negative. 


8 


7 


5 


Some materials have mistakes. Maths material in item pools is useless. 
Maths tests are inappropriate. 


3 


6 


0 


Cause of conflict between NTTF (hence teachers) and Department. 
Suspect this is a step toward teacher assessment 


3 


0 


1 


Administration is not uniform Teachers teach to the tost/hfiln ktri^ 
with the tests. 

Totals 


28 


A 

26 


11 


Not answered, no comment. 


6 


18 


1 0 


Total number of respondents 


38 


5 2 


5 2 
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In response to the questions concerning participants' attitudes 
towards the program as a whole, several of the responses did not 
fit into the categories in Table 8, As these varied greatly and were 
distinct from any other comments, some examples are provided 
below: 

/ would dearly like to see scholastic ability rewarded on 
an individual basis by the introduction of a Year 7 
Scholarship which would give parents/students/teachers 
incentive to pursue even greater levels of academic 
excellence. 

Maths tests are too small and 'bitsy'. Children are 
confronted with a 'paper war' on their desk. 

Tests are time consuming. Teachers and students are not 
committed or motivated to perform well. Tests (reading) 
are culture-biased. Errors are contained in many of the 
tests and have not been corrected in the past three years 
of my experience teaching Year 5. 

I would like more explanation of WHY system-wide testing 
(examinations) are needed: their purpose? Why not 
continue with sample testing for curriculum and standards 
purposes? 

If you want standardised results then the tests will have 
to all be administered the same way, otherwise you are 
wasting your time. Hopefully the Maths test will do what 
you are claiming and not have a bias to a particular area of 
Maths. 

Parents are not happy with the assessment program 

Assessment of children's learning is essential but NOT on 

a system-wide style basis A recent survey of our 

parents say they are not anxious to have external exams 
for primary age kids. The failure syndrome is not 
conducive to effective teaching and is inherent in system- 
v/ide tests. 



Please keep material and criteria for Aboriginal schools 
separate from urban schools at this stage. 



Test twice a year so teachers can see if tfiere tias been an 
improvement in their class. Tests could be handy for 
teacher's information and maybe school information but I 
don't think they should be used to compare schools. 

For the testing program multiple copies should be sent to 
schools. - - 

Make a video explaining the materials and how to use them 

Need to compare NT students with students from other states. 

As can be seen, these comments are wide and varied, however, there 
are some ideas that may be worth pursuing in any further review of 
the Primary Assessment Program. 



4.7 Teachers' Views on Test Administration and Timing 
of the Tests 

The major concern of schools with test administration, which was 
evidenced by comments on the questionnaires and during 
interviews, was the amount of photocopying that was required to 
prepare the tests and the time taken for administration. Changes 
in 1988 and 1989 should eliminate most of these problems. 

The administration booklets sent with the 1988 test materials 
were thought to be an improvement on the previous instruction 
information. However, some teachers felt there was too much 
general information at the beginning of these documents. The 
following are examples of comments related to test 
administration: 

In the document 'Northern Territory Primary Assessment 
Program for Urban Schools - Guidelines for Test 
Administration 1988' there is a lot of reading before you 
actually get to the instructions for administering the 
tests. 

The guideline booklet is good.' It is clear and 
• straightforward, better than last year. 
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It would be good to have what group the school is in 
listed on the front of the book with the '^chool number. 

' There is no information in the book about what we do ' 
with kids who are absent when the tests are given. We 
have tried to administer the tests when these kids return 
to school. 

We had problems administering the new Mathematics test 
(for Aboriginal schools). This 65 item test is printed on 
both sides. A total of seven sheets of paper were used 
stapled in the top left hand corner. The fact the test was 
stapled at the top meant that when students had 
completed one page of questions they had to lift the 
whole paper, turn the previous page over the staple and 
• turn the whole paper over to do the next set of questions. 
This caused some students to miss whole pages of 
questions and to become irritated. If the test had been 
made into a booklet with two staples on the left-hand 
side this problem could have been eliminated as children 
could then Just turn the pages like a book. 

The NOTE TO TEACHERS on the test said that it was to be 
administered in two sessions of 40 minutes each. 
Administration would have been made easier if the test 
had been put into two booklets. 

The answer sheets would be better if they only had the 
answers for the particular test and not other tests that 
the student is not required to do. 

Teachers were asked to provide information as to the most 
suitable time to administer the tests. A range of times and 
suggestions was provided, ranging from the beginning of the year 
to the end of the year with suggestions ranging from not 
administered at all to administered every term. The following 
table summarises the replies for the three groups of respondents: 
urban seniors (US); urban teachers (UT); and staff in Aboriginal 
schools (AS) 



Table 9 Timing of the Tests. 





y§ 




A§ 


End of Term 3. (September) 


1 0 


9 


1 7 


End of year. November. 


12 


3 


7 


Beginning and end of year. End of each Semester. (Twice per year) 
Once per term. (Four times per year) 


6 


11 


5 


Beginning of Semester 2 (August) 


1 


8 


7 


Mid Term 4. (October) 


1 


5 


3 


Never. 


7 


2 


8 


Total number of comments 


37 


38 


47 


Question not answered. 


2 


0 


1 0 


Total number of respondents 


38 


52 


38 



•It appears that September is the most favoured time for test 
administration, however, the end of the year was also a favoured 
time, especially with urban seniors. From discussions with 
teachers and comments in other sections of the report it appears 
different times are seen as more appropriate for different tests 
and for different school groups. The following comments 
elaborate on this theme: 

V^hy was Year 7 chosen? If given at the end of Year 6 or 
at the beginning of Year 7 we could find out what kids 
l<now and need for preparation for high school, (urban 
teacher). 

November is a very bad time in Aboriginal schools to 
administer tests. Suggest late third term with results to 
schools fourth term, (teacher in an Aboriginal school). 

It would be a good idea to stagger the tests. For example 
the reading tests couid be administered at the beginning 
of sSemester 2, reading levels won't change. The results 
can be returned to schools within a month or so. This 
would allow schools to compare their classes with the 
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rest of the Territory and give teachers a focus for their 
programming for the rest of the year. The Year 7 
Mathematics test could be given at the end of the year 
when teachers had completed the course. Results could 
be used by secondary schools for placement of pupils, 
(subject adviser). 



There is concern about the timing of the tests. The 
middle of October is too early - Year 7s have not finished 
their course. Results are not useful for high schools as 
they are not a true indication of what children can do at 
the end of primary school. Testing should be for a 
purpose. If the program is to provide an indication of 
standards at the end of primary schools then it should be 
administered at the end of primary schooling. To be 
useful to primary schools the testing program should be 
diagnostic. It would be very useful for schools to have 
tests aimed at the end of Year 6 level which could be 
administered at the beginning of Year 7. This could 
provide diagnostic information for the school and also 
provide Territory wide information which could compare 
schools. The Year 7 tests could be administered at the 
beginning of the year to Year 8 students. This would 
serve two purposes - it would provide the Department 
with the information required as to the academic 
standards in Maths and English of students who had 
completed primary school and also provide high schools 
with placement information for their students (compared 
Territory wide). It would also be good to have a test at 
the beginning of Year 4, (principal, urban primary school). 

The comments provide conflicting views as to the most suitable 
time for test administrastion. One of the determining factors for 
the differing views is the purpose for which teachers see tests 
being employed. There appears to some confusion amongst staff as 
to the diagnostic nature of the Primary Assessment Program. It 
appears that teachers only consider tests can be diagnostic on an 
individual level and do not view the testing program as being 
diagnostic on a school or state level. Possibly, teachers need to be 
made more aware that the Primary Assessment Program materials 
can be used diagnostically by schools or individual teachers at any 
time throughout the year. 
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4.8 How Teachers Became Aware of the Program 



Staff were asked to comment on how they found out about the 
Primary Assessment Program. The following chart is a summary of 
replies from urban seniors (US), urban teachers (UT) and staff from 
Aboriginal Schools (AS). 



Table 10 Finding Out About the Primary Assessment Program 





1 1 o 


t i T 

JUL 




btan rneotmgs. rrom oiner siarr. 


Q 
O 






Departmental circulars. Receiving materials in the school. 


1 9 


1 0 


22 


Through participation in the testing program. Told to do the tests 


■ 9 


12 


■ 9 


Involved in trialling. Involved in materials development. Through media. 


2 


4 


1 


Inservice at the school with the writing moderation. 


3 


4 


2 


Didn't know it existed. Through this questionnaire 


1 


3 


4 


*Totai number of comments 


42 


56 


48 


No comment 


6 


1 0 


1 5 


Total number of respondents 


38 


52 


38 



•As some of the respondent's comments involved more than one point in relation to the way in 
which they found out about the program, the number of comments is greater than the number 
of respondents. 

The table indicates that most of the teachers found out about the 
program either by contact with other members of staff or through 
Departmental circulars. The next most common method of finding 
out about the program was by participation in the testing program. 
A number of school staff, especially in Aboriginal schools did not 
make a comment or said they did not know the program existed, e.g. 
50% of staff from Aboriginal schools fitted into this category. 

Durii.g interviews with school staff, it was cbviojs that many did 
not know about all of the materials. The degree of ignorance was 
such that the program had to be explained and materials shown so 
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that the interviewees understood what the interview was about. 
Most participants, if they knew anything about the Primary 
Assessment Program, understood it to involve the tests that were 
to be administered at the end of each year. Several principals did 
not know what materials were supposed to be in the school. 

After explanation and display of materials the majority of 
interviewees expressed interest in using the materials and learning 
more about them. Examples of comments which demonstrate this 
include: 

As far as inservicing the program, this should be aimed at 
prinoipals, assistant principals and band 2's at the 
beginning of each year. In places like Darwin, advisers 
could attend school meetings or joint school meetings to 
give information on the program. 

The idea behind the materials must be 'sold'. Firstly to the 
assistant principals and seniors who must then 'sell' it to 
their teachers. 

There is a lack of knowledge and understanding of the 
program. It needs to be explained to 'senior management'.. 

A video showing the materials, and explaining the testing 
" program would be very useful to schools. It would be 
great for old and new teachers alike. 

The materials appear to be useful resources. All upper 
primary teachers should know about them. 



4.9 Achievement of Program Aims: Views of Senior Staff 

Principals and seniors in urban and Aboriginal schools were asked 
whether they felt each of the two basic aims of the Primary 
Assessment Program were being met. The two aims are: 

1, to monitor achievement in English and Mathematics in Northern 
Terr' ory primary schools. 
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2. to provide teachers in the upper primary with materials for 
assessing the competency of students in Mathematic j and 
English. 

A total of 38 questionnaires were received from urban seniors (US) 
"and 27 from seniors "in Aboriginal schools (SA). Comments on each 
aim were analysed into two categories - agreement (Yes) 
disagreement (No). A summary of replies- is shown in the following 
tables. . ' 



Table 11 Achievement of Aim 1 



Aim 1 (to monitor standards} 
Yes (Agree) 


v$ 


SA 






Yes, feedback indicates this. Gives general indication of standards. 


4 


1 


Partially - the package can assist teachers in assessing some facets 
of competency. At a basic level. Reading partially, not enough feedback. 


6 


5 


Yes (with no comment supplied). 


9 


2 


Totals 


19 


8 


No (Disagree) 

No, to test minimum competency is not an indication of the standard of 
achievement Results unreliable as teachers assist students. 


7 


4 


No, philosophically opposed as they do not benefit children. NTTF bans 
No, not at present. Tests are limited in some areas. 


1 


4 


No, (with no comment supplied.) 


1 1 


8 


Totals 


19 


1 6 


Not answered Totals 


0 


3 


Total number of respondents 


38 


27 



Half of the seniors in urban schools agreed that the aim of 
monitoring standards was being achieved while the other half 
thought the aim was not being met. Only 8 (33%) of the 24 seniors 
in Aboriginal schools who answered this question, thought that aim 
1 was being achieved. Of those who did agree, many thought that it 
was only partially being achieved. The main reasons given for the 
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aim not being achieved were related mainly to the tests and test 
acifninistration. . 



Table 12 Achievement of Aim 2. 



Aim 2 (Assessment materials for use In schools) 


V5 


5 A 


Yes (Agree) 

Materials are quite sound. Used within schools as a means of monitoring 
effectiveness of instruction. Generally useful. Writing moderation excellent. 


13 


4 


Yes partially. Although 1 don't think teachers make as much use 
of this as they could. Not all of the material is suitable. 


4 


5 


Yes ( with no comment supplied.) 


7 


9 


Totals 


24 


1 3 


No (Disagree) 

Not utilised by teachers. Teachers do not use materials. Teacher prepared 
tests based on actual teaching is more relevant 


4 


3 


English no. 


2 


0 


No. (with no comment supplied) 


10 


2 


Totals 


14 


5 


Not answered Totals 


0 


4 


Total number of respondents 


33 


27 



The majority thought the aim 2 was being achieved and that the 
materials were useful to teachers or they felt that they could be 
useful. 



Over 60% of senior staff in both urban and Aboriginal schools felt 
that the program had achieved the aim of providing assessment 
materials for teachers to use. Of those who disagreed or felt that 
the aim was partially being achieved, the reasons provided focussed 
on the point that teachers do not utilize the materials. 



53 

Er|c 6 J 



Approximately half of the participants, both from urban and 
Aboriginal schools, provided a yes or a no answer without making a 
comment. 



4.10 Improving Academic Standards? 

Teachers and seniors in urban primary schools and Alroriginal 
schools were asked if they felt the Primary Assessment Program 
was helping to improve academic standards across the Territory. 
As with the achievement of aims, replies were varied. The 
following table summarises the replies. 



Table 13 Improving Standards? 





US 




A9 


Yes (the program helps to Improve academic standards) 








It sets a standard. It motivates teachers and students, it is a challenge for 
those not doing well. Any form of assessment must help improve standards. 


8 


10. 


13 


The new year 7 Maths test and the writing moderation booklets have 
improved standards. Is a starting point need further improvement 


1 


6 


6 


Plays a diagnostic role. 


2 


1 


3 


Totals 


11 


17 


22 


No (the program does not help to Improve standards) 








There is little feedback from the Department. More follow-up is required. 
So little known about it Lack of awareness. 


9 


7 


4 


Only used for system-wide tests. It is a too one-offish event Schools 
teach to the test and thus invalidate results. 


2 


5 


2 


This type of testing is too restrictive. Curriculum is not rigid enough 
to allow such rigid testing. Not diagnostic. Chore for teachers. 


3 


4 


4 


4 

Tests cannot improve standards. Are of no benefit to the school. ■ 
Not relevant Not reliable. Not useful for ESL students. 

Totals 


1 

1 5 


4 
2 0 


6 

1 6 


Total number of comments 


2S 


3 7 


3 8 


Not answered or unsure Totals 


1 2 


1 5 


1 4 


Total number of respondents 


3 8 


5 2 


5 2 
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Approximately 60% of the urban teachers and seniors who answered 
this question felt the Primary Assessment Program did not help to 
improve academic standards. This figure was reversed for staff in 
Aboriginal school where approximately 60% of those who commented 
felt that the program was assisting in improving academic 
standards. 

The main reason provided for improvement was that the program 
sets a standard and provides motivation for students and teachers. 
Reasons given to substantiate why the program was not improving 
standards were; a lack of awareness of the program; and that the 
testing was too restrictive and not diagnostic. 



4.11 Use Made of Information by Schools 

Personnel were asked if they had seen the results and reports and 
to what uses this information was put. Tables 14 and 15 provide a 
summary of replies to this question. 



Table 14 Awareness of Results 



YES havs saan results 

Usa to evaluate class weaknesses and strengths. 
Compare with Territory average. 
Of little value, not usee'. 

Not followed up as results came too late. Results too technical. 

Tota 


U5 


UT 


AS 


1 7 

7 
1 1 

5 

la 33 


5 
4 
3 
2 

25 


7 
3 
1 1 
2 

25 


NO not seen Totals 


2 


2S 


2 2 


Question not answered Totals 


3 


2 


5 


Total number of respondents 


38 


32 


5 2 



Eightly seven percent of urban seniors had seen the test results 
while only 50% of urban teachers had seen the results. Fifty 
percent of staff in Aboriginal schools had seen the results. 
Seventy three percent of the urban seniors who had seen the 
results thought they were useful to evaluate strengths and 
weakness and/or useful for comparisons with NT standards. 
Thirty three percent of urban seniors and 44% or staff from 
Aboriginal schools who had seen the results did not use them or 
thought they were of little value. 



Table 15 Awareness of Reports 





y§ 


VT 


A? 


YES have seen reports 








Take note of strengths and weaknesses. Teach where results are low. 
Good to know patterns of student performance for the NT. 


1 0 


s 


6 


Of little value. Well presented and documented but not used. 


13 


4 


8 


Would be good to get reports earlier. 


1 


6 


2 


Totals 


26 


2 1 


2 1 


NO have not seen reports 








No (with no comment provided) 


1 0 


21 


22 


Never. What reports. Didn't know there were any 


0 


2 


4 


Totals 


10 


2 3 


2 6 


Question not answered 


2 


8 


5 


Total number of respondents 


28 


52 


52 



Seventy one percent of urban seniors, 40% of urban teachers and 
40% of staff from Aboriginal schools had seen the reports. Of 
those who had seen the reports approximately half of these 
thought that although the reports \.'ere well presented, they were 
of little value to the schools. Respondents who had seen the 
reports thought them useful in evaluating weaknesses and 



strengths and in comparing their students with Territory 
standards. 



4.12 Use Made Of Information by the Department 

The use made of the information gained from the Primary 
Assessment Program, and in particular from the testing program, 
was not a major concern of this study.. However, during visits to 
schools and in discussions with Departmental personnel, several 
references were made to the perceived lack of action on the test 
results. 

The following comments are examples of this concern: 

Who in the Department looks at the results? If they are 
looked at, what is being done with the information. 
Someone should follow this up (Education Officer). 

It would be useful if results could go to schools at the end 
of the testing year and that maybe the Territory results 
could be included in individual report cards The results 
are not being used to their full advantage (principal, urban 
school). 

I am concerned that schools have the option of choosing 
who should sit the tests, this skews the results. All 
students should sit the tests and results should be 
available fo~r advisory staff to assist schools in areas of 
weaknesses (superintendent). 

What happens to the results? The Department does nothing 
with the results anyway. Schools and teachers do not get 
any more assistance or inservice (teacher, Aboriginal 
school). 

The testing program must change with changes in 
curricula. The English tests should be kept in line with the 
new English curriculum. With the change in emphasis 
towards teaching of different genres test scripts should' 
reflect each of the genres. The topics and information in 
the tests must be up-to-date especially if it comes from 
such sources as newspapers (curriculum officer). 



The area of upgrading materials and tests is being addressed by ttie 
Primay Assessment Committee by way of continual review of the 
program. However, action to improve test results is a much broader 
issue and under current policy individual school results are not 
available to Departmental personnel. In the original agreement on 
the Primary Assessment Program all results are confidential: 

It is therefore imperative that such tests be devised to 
encompass the widest scope of the courses and secondly, 
that confidentiality of the results of schools and teachers 
be maintained (Minutes of the Curriculum Advisory 
Committee Meeting, December 1982, p. 23). 

This has been advantageous to the success of the program in that 
schools have participated because results are kept confidential. 
However, this has also meant that superintendents/subject advisers 
do not know which schools do badly and cannot take appropriate 
action and cannot identify, where performance is good and examine 
factors in such schools which could be applied to other schools to 
improve performance. 

The results are published in the annual reports which describe areas 
of strength and weakness in Mathematics and English at the levels 
tested. These reports are available to all Departmental personnel. 
Since 1987 the individual school results included final scores and 
percentage scores for each student on each test as well as the 
school result and the Territory result. This should have allowed 
schools to use the results to work on areas of weakness, or to 
commend teachers on areas of strength in the school. 



4.13 The Year 7 Mathematics Test 1988 

Because the new Year 7 Mathematics was very different from the 
previous tests, a separate evaluation sheet was developed and sent 
to schools for information on the reaction to this test. Questions 
were asked on the time allowed, date and format of the tests as 
well as the range and difficulty of items and their relation to the 
curriculum. Participating schools were asked for further 
comments on the Year 7 te^t or on the Primary Assessment 
Program. Table 16 summarises the replies to this questionnaire. 



58 



6o 



Table 16 Reactions to the Year 7 Mathematics Test 1988 



• 

Tim© allowed for test ( La. 75 minutes) 


N 


% 

- 


Time allowed for test is satisfactory. " " 


22 


61% 






1 1 % 


Not enough time was allowed. 


7 


20% 


Divide tine test into two sessions. 


3 


8% 


Total number of comments 


36 




Date of the test (19th October 1988) 






Date satisfactory. 


5 


1 9% 


Too early, Year 7 work incomplete. 


8 


31% 


i oo laio e.g. Tor Qiagnosuc purposes. 


9 


35% 


Should allow more flexibility. 


4 


15% 


Total number of comments 


26 




Format 






Format satisfactory. 


19 


70% 


rormajiiy ana lengin orr-puning. otuoents panic at formality of the test. 


6 


22% 


Fewer multiple choice would be better. Items could be grouped. 


2 


8% 


Total number of comments 


27 




Range and Difficulty 






Satisfactory. 


1 3 


34% 


More basic questions needed. Too difficult 


13 


34% 




y 




Too little emphasis on problem solving. Include harder questions. 


3 


8% 


Total number of comments 


38 




Relation to the Cun'fculum 






Satisfactory/com pre hensive/ralevant. 


1 0 


63% 


aeyoncj wa iviainemaiics syiiauus ai year /. 


4 


25% 


The syllabus stresses decimals e.g. for tc. 


2 


12% 


Total number of comments 


16 




Further Comments 






Year 5 should have a similar test to reduce photocopying. 


3 


14% 


New test is more suitable. 


4 


1 9% 


More tests of this type are needed, e.g. at year 6 as a pre-test. 


5 


24% 


Students with learning difficulties suffer. 


3 


1 4% 


Other. 


6 


29% 


Total number of comments 


21 




Total Number of Respondents 


43 
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Sixty six schools were contacted and 43 questionnaires were 
returned. Where multiple comments were made they were counted 
in the most appropriate section. 

Over 60% of respondents felt the time allowed for the tests, the 
test format, and the tests' relation to the curriculum were generally 
satisfactory. However, there was disagreement on the date of the 
tests, with the greater proportion of teachers saying the test was 
either too early because Year 7 work was not complete, or too late 
to be of value diagnostically. Quite a number of teachers felt that 
more basic questions were needed and that some of the questions 
were ambiguous or were too difficult in that they tested a range of 
concepts in the same question. 

Comments such as 'Year 5 should have a similar test' have already 
been taken into consideration by the Primary Assessment 
Committee which has proposed a year 5 test based on the Board 
Approved Curricula for the 1989 testing program. 

Other common comments included: 'the new Year 7 test is more 
suitable', 'more tests like this are needed e.g. at a Year 6 level or as 
a pre-test for Year 7'. 

Common negative comments included: 'students with learning 
difficulties and English as a Second Language suffer*, 'students 
panicked at the formality of the test'. 
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5.0 SUMMARY, CONCLUSIONS AND ISSUES 



5.1 Introduction .............. ..^ ...... 

In summary the two major aims of the program were to; 

(i) monitor academic standards; and 

(ii) provide assessment materials 

During the period 1984 - 1989 a total of 10 assessment documents 
have been produced and introduced into schools. During the 
introduction of the program in 1984 into urban schools there was a 
high level of inservice provided. In 1986 when the program was 
introduced into Aboriginal schools, very little inservicing was done. 
Subsequently there was little follow-up inservice on the program 
prior to that conducted in 1989. 

The aim to monitor standards appears to have been achieved, 
especially with the urban program. However, due to union bans and a 
lack of knowledge of the program by staff, the participation rate for 
Aboriginal schools has been lower than expected. 

A survey of teajher perceptions of the program was carried out by 
questionnaire and a number of schools were visited. Completed 
questionnaires were received from: 

38 senior staff in urban primary schools (57% response rate); 

52 Year 5, 6, 7 urban primary teachers; 

52 staff from Aboriginal schools (37% response rate); 

43 urban school staff who were involved with the 1938 Year 
7 Mathematics Test (65% response rate). 



5.2 Summary of Findings 

Awareness and Use of Materials 

Approximately 70% of all staff had seen some or all of the materials 
and approximatly half of all staff had used at least some of the 
materials. 
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The most commonly used materials were the Mathematics item pools 
which were mainly used for system-wide tests and the least used 
was the Type C document 'Strategies for Reading Assessment' 

Seniors in urban sci-iOols were the group most familiar with most of 
materials with urban teachers being the group least familiar with 
materials. _ • . _. . 

Approximately 80% of seniors had seen the materials and 
approximately half of these had used the materials at some stage. 

Approximately 50% of Year 5, 6, 7 teachers in urban primary schools 
had seen the various assessment materials, with fewer having used 
these, especially in 1988 

Approximately 80% of staff in Aboriginal schools had seen the item 
pools and the writing moderation booklet but only 43% had seen the 
Type C document. Use made of the materials, however, was low, 
especially use in 1988. Again the greatest use of materials occured 
with the item pools which were used for system-wide testing. 

Materials - Their Place in Schools 

Based on visits to school, there appears to be a degree of ignorance 
as to what the Primary Assessment Program is. Many of those 
interviewed perceived the program to involve only the annual testing 
and were not aware of the assessment materials, and if they were 
aware of the materials e.g. the writing moderation booklets, they did 
not realize these were part of the Primary Assessment Program. 

Strengths and Weaknesses of Materials 

The item pools were the most commonly used materials and the 
perceived strengths were that they provided a standard and were 
useful reference materials for teachers. The perceived weaknesses 
were that they were not related to classroom learning, the 
Mathematics item pool is too simple for the year levels for which 
they were prescribed and that the preparation of tests from the item 
pools was very time consuming and a waste of paper. 

The writing moderation booklets were very popular. Both the 
teachers from urban and Aboriginal schools felt these provided very 
good support material and allowed comparison of standards of their 
students' writing with the rest of the Territory. There were very 
few comments on weaknesses of the writing moderation booklets. 



o 

ERIC 



62 



Only a small percentage of teachers commented on the type C 
document. However, those who did comment perceived it as a good 
resource providing a variety of ideas and activities. One of the 
weaknesses mentioned,, which does not relate to the intrinsic, merit 
of the document, was that it needs to be promoted more. 

Strengths and Weaknes sfts of tha Program 

The main strengths of the program are that it provides a standard and 
useful reference materials. 

The most common weaknesses perceived were; lack of relevance, 
especially for Aboriginal children, mistakes in some of the 
materials, late feedback on results and wasted time and money in 
administering the tests. It was also mentioned that the program was 
a cause of conflict between the teachers' union and the Department. 

Attitudes Towards the Program a s a Whole 

Comments were evenly divided for and against the program and there 
was a wide range of reasons to substantiate the two views. 
Generally, those in favour thought it worthwhile because it provided 
a standard and useful resource material while those against thought 
it a waste of time and money, and not necessary. 

• Test Administration and Timing of the Tests 

Major concerns were the amount of photocopying and the time taken 
for test administration, although comments concerning the 1988 Year 
7 Mathematics Test indicated that sending multiple copies of test 
materials to schools was a much appreciated change. 

September was the most favoured time for testing, followed by the 
end of the year, however, there were many suggestions for 
administering different tests at times appropriate . to the subject and 
year level. 

• How Teachers Became Aware of the Program 

Most commonly, teachers became aware of the program either though 
contact with other members of staff or through Departmental 
circulars. However, a large proportion, e.g. half of the staff from 
Abroginal schools, did not answer this question or said they did not 
know they program existed. The perception that many teachers had. 
little knowledge of the program was sustantiated during visits to 
schools and in interviews with staff. However, suggestions on ways 
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to explain the program through inservice and/or producing a video 
were well supported. 

Achievement of Program Aims . , 



Aim 1 

Fifty percent of the seniors in urban schools agreed that the aim of 
monitoring standards was being achieved while the other 50% 
thought the aim was not being met. Only 33% of seniors in Aboriginal 
schools thought that aim 1 was being achieved. Of those who did 
agree, many thought that it was only partially being achieved. The 
reasons given for the aim not being achieved were related mainly to 
the tests and test administration. 

Aim 2. 

Over 60% of senior staff in both urban and Aboriginal schools felt 
that the program had achieved the aim of providing assessment 
materials for teachers to use. Of those who disagreed or felt that 
the aim was partially being achieved, the reasons provided focussed 
on the point that teachers do not utilize the materials. 

• Imorovina Academic Standards? . 

Approximately 60% of the urban teachers and seniors felt the 
Primary Assessment Program did not help to improve academic 
standards. This figure was reversed for staff in Aboriginal schools 
where approximately 60% of those who commented felt that the 
program was assisting in improving academic standards. 

Reasons given to substantiate why the program was improving 
standards were; that it sets a standard and provides motivation for 
students and teachers, Reasons given to substantiate why the 
program was not improving standards were; a lack of awareness of 
the program; and that the testing was too restrictive and not 
diagnostic. 

• Awareness and u.qe nf Resnits and Reports 

Eighty seven percent of urban seniors had seen the results, whereas 
only 50% of urban teachers and staff in Aboriginal schools had seen 
them. Even smaller proportions of school staff had seen the reports. 
The urban seniors felt the results and reports were useful in 
monitoring strengths and weaknesses in their school and in providing 
a Territory ^'andard. The majority of urban teachers and staff in 
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Aboriginal schools felt the results and reports were of little value. 
The 1988 Year 7 Mathematics Test 

Staff in urban primary schools were generally satisifed with the new 
Year 7 test. The majority felt it was a vast improvement on the 
previous Phase 3 Mathematics tests. 

The Testing Program as a Whole 

The testing approach has been continually changing and developing 
over the operation of the program. Major changes occurred in 1988 
and are proposed for 1989. These alterations should alleviate the 
problems schools have encountered in photocopying materials and 
administering tests. 

The 1988 Year 7 Mathematics tests was generally thought to be 
satisfactory in format and content. 



5.3 Conclusions 

For the program to be effective as a whole, those involved including 
principals, seniors, subject advisers and regional superintendents, 
must be aware of the program as a whole. During interviews and in 
interpreting most of the answers to the questionnaires, the 
comments made were directed at the testing program rather than the 
program as a whole or . at the assessment materials. The overall 
impression is that the majority of school staff are ignorant as to the 
purposes of the assessment materials. From interviews this was 
very apparent and it is suspected that some of the comments about 
the materials may have been made in ignorance. On the other hand, 
teachers who had used the materials as adjuncts to teaching were 
obviously impressed and found these very useful. A similar 
observation was made in regards to test results and reports. Those 
principals and seniors who had marked tests done at their schools, 
analysed results and made note of differences with Territory results 
saw the testing program as being valuable. 

One of the main reasons it appears that the materials are not used as 
planned is that they have been held for the annual testing. As 
mentioned earlier in the report, the current and proposed changes 
will eliminate the need for the item pools to be 'kept' for the annual 
testing. The problem however is that teachers currently in the 
schools are not aware of the materials, and also it is suspected that 
materials have been misplaced or lost and that schools no longer 



have the 'full set'. The current writing moderation bool<lets have 
been well received, but are not generally i<nown to be part of the 
Primary Assessment Program. 

The Primary Assessment Program has been successful in providing - 
factual information about standards of student achievement in 
English and mathematics at Year 5. and Year 7. . It has also made 
available a range of assessment materials tor teachers, but the 
potential of both of these and the information about standards has 
yet to be fully realized. However, the testing program has undergone 
several changes over the last two years which should help to counter 
some of the criticisms of the program recorded in this report. It is 
essential that new curricula and changes in approaches to teaching 
be considered in further review of the tests. 

The majority of seniors in urban primary schools appear to be aware 
of the materials but the level of productive use has varied greatly. 
In general, teaching staff in both urban and Aboriginal schools are 
not aware of the full potential of the assessment materials and this 
appears to have limited the use made of them. 

For the program to be successful as a whole there is a need to ensure 
increased understanding of its potential and greater communication 
between the Department and the schools. The last two years have 
seen a move in this direction. 



5.4 Issues 

(i) Many teachers have little understanding or knowledge 
of the Program. 

This suggests the need fop an ongoing campaign to communicate the 
intentions and content of the Program to staff in schools However, 
the methods of promotions must be carefully considered and all 
options explored, to achieve the most cost-effective approach. 

In 1989, two inservice courses on the Program were organised, to 
cater foi- urban schools in Darwin and Alice Springs. The level of 
response suggest'-d that such courses may be worth repeating. There 
is also the need to extend such opportunities to staff in Aboriginal 
schools. Cost, however, is lit<ely to be a limiting factor. 

A brochure could be produced and sent to schools explaining the 
program and describing materials, in addition to the general 
circulars that go to schools. Case studies of schools where the 



program is being used effectively could also be of value in 
demonstrating its potential to other schools. 

A popular idea and one that might cater for all teachers requiring 
information on the program could be the making of a video which 
would give background information on the., program, describe fully 
how to use the materials, and allow schools to understand results. 

In addition, supervisory and advisory staff could discuss with 
teachers the use made of materials and results, during their routine 
visits to schools. There is the question of the repsonsibility of 
teachers to make themselves aware of what is available and its 
potential use. There must be a two-way responsibility. Without 
this, all the efforts of the Department are likely to be in vain. 

(ii) Some schools have incompiete sets of materials 

In view of the changing direction of the Program, it may be 
inappropriate to consider reissuing the original materials, either in 
their present, or in a revised, form. However, items from the new 
style tests will be available to schools and will gradually 
accumulate to form item banks which will replace the original item 
pools. Although this is a slow process, it is the optimum achievable 
within current resources. 

Similarly, the production of writing moderation booklets, urban and 
Aboriginal schools in alternate years, will continue. Again, this 
approach is achievable within current resources. 



( i i i ) Teachers have expressed doubts about the relevance of 
certain Primary Assessment Program materials for 
classroom use. 

Because the items were targeted at the core objectives, they were 
frequently seen as too easy for many upper primary students. 
However, schools are free to use the items at whatever levels they 
consider appropriate. The range of items in the new tests, which are 
designed to assess not only the core objectives, but also the 
Approved Curriculum, should overcome this problem. 

(iv) Only about half of all Aboriginal schools participate in 
the Program. 

Of the non-participants, approximately half give as their reason the 
fact that none of their students is achieving at a Year 5 level. The 



remainder give no reason for non-participation since they do not 
respond to communications about the Program. 

There is thus clearly a need to achieve greater participation in the 
Program by Aboriginal schools." Some possible methods to 
accomplish this may be to hold sessions at as many inservice courses 
as possible that involve staff from Aboriginal schools, visit schools, 
and/ or get advisers who are visiting schools to discuss the program 
while they are there. So far, all efforts have aimed at encouraging 
teachers to carry out their duties. This could be insisted upon. 

It may also be desirable to reconsider the current policy which 
allows Aboriginal schools to exclude any students not considered to 
be performing at the Year 5 level. An alternative would be to adopt 
the rules which apply in urban schools, where only students in 
special classes are exempted. Also, the range of difficulty of the 
tests could be extended, particularly at the lower end so that 
virtually all children can achieve some success. 
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Appendix I 



APPENIJ1CE5 



1.4 PROCEDURES FOR IMPROVING ITEM QUALITY 

There are several procedures which can be used to identify items that ar 
'faulty'. In order for the test to have content validity, the items mus 
fit. the objective being measured. This was done by inspection. Unless on 
can say with confidence that the items in a competency test measure th 
intended objective, any interpretation of the test scores is questionable. 



FACILITY 

The facility of an item was a very important characteristic used in 
selecting items for the program. This is indicated by the proportion of 
students who got the item right- 

EXAMPLE 

If in a class of 20 students, the number who got an item right was 12, the 
facility of the item is given by: 

Facility * Number who cjot the item right 

Total number who attempted 

20 

= 0.60- 



Facility can have a range of values from 1.00 where everyone got the item 
right to 0.00 where no one got the item right. 

The facility or p-value was a useful indicator for analysis. In the CR 
test situation, high p-values could be expected if what was measured by the 
items had been taught well. A high or a low p-value did not mean a oad 
item. If the p-value was high or very low, the item was checked to see if 
it discriminated betv^een mastery and non-mastery students. 

A good way to use the facility value in a class situation is to obtain one 
before instruction and another after instruction. If, however the p-valuss 
were close, one may examine the instruction or the test items. 
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DISCRIMINATION 



'Another characteristic used in the analysis was the item's discrimination 
index. -Since- the aim of a CRT is to.teM the. teacher whether-the student- 
is competent or not on a particular objective, it was important to select 
- items that discriminated -between students who were competent (masters) and 
those who were not (non-masters). 

An Item discriminates if a higher proportion in the competent group 
(mastery group) got the item right compared with the proportion in the non- 
competent group (non-mastery group) who also got the item right* The 
competent group comprises students who have obtained 80% of the number of 
items correct where 80% has been defined as the competency level. 

The discrimination index is given by: 

Discrimination Index - No of students in mastery group who got item 

right 



Total no of students in mastery group 

No of students in non-mastery group who got 
item right 



Total no of students in non-mastery group 

EXAMPLE 

In a class of 20 students, 14 students achieved the mastery level of 80^ 
correct on a test. This means that 

No of students in mastery group » 14 

No of students in non-mastery group = 6 

Suppose, on. a particular item in the test the number of students in the 
mastery group who got the. item right was 12, and the number in the non- 
mastery group who also got the icam right was 2. 



^3 «3 



Then, the discrimination index is given by: 



Discrimination Index 



12 



2 



14 



6 



0.86 



0.33 



0.53 



A discrimination index of 0.53 indicates a highly discriminating item. A 
low value such as 0.10 or less, or a negative value, is a warning flag. 

When a seemingly poor item was located, one had to decide whether to alter 
or discard it. Other characteristics of CRTs were also examined, but these 
would take too much space to explain here. The panels involved in the 
development of the item pools for this program selected items that had 
certain characteristics that 'good' CRTs possess. 



74 



Appendix II 

ADAPTED LEVELS OF USE INTgRViEW SCHEDULE FOR SCHOOL STAFF 
For Teachers 

1. Are you currently using materials from che program? 
Here Is a picture of the materials that nave oeen prccucea. 
Which ones are you currently using. 

<If the answer is no, asK if they know aoout tne program anc 
prode what they know-) 

2. Please aescrice for me how you use tne materials. 
(Probe their use of the three strategies - Item Pcois, 
Moderation book lets, and Type C) 

OR for PrlncLoais ana S enior Teachers . 

1. Is your school currently using materials from tne 
Primary Assessment program? 

2. Please descrice for me how the materials are usea. 

3. What do you see as the strengths and weaknesses of tne 
program in your situation? (Prcce for strengths anc 
weax.ness in the three types of materials - Item; ?cc;3. 
Moaeration. and Type C) 

4. Hew aid you find out accut the Primary Assessment 
Program? 

5. Do you meet wi^h c:her to ciscuss tr.e program? Vnat 
sorts of things are tal<eG aoout? 

6 . As you 1 ock ahead th : s year . what p : ans cc ycu r. f /e . n 
relation to your use .of the primary assessm^ent program/ 

7. Do you think the iducationn Cepartm.en: sp.cu.c : ry 
estaclisn systemi-uice scanocrcs? 



BEST copy AVAIL/IBLE 



The materials shown Del ow have been sent to. your school over che past- 
four years. Have you seen these materials? Please answer l£s or lis. in 
the tjoxea provided. Please Indicate by ticking <✓') the boxes tnose 
materials' that have been y^^^n this year or have been »jSg<3 in ';ng 
Please indicate using A, B, C, or D, what each of the macerials is/was 
used for . (Note: they may have more than one use) 



system-wide tests selected from the- item pools 
class teats or exercises selected from the Item pools 
assessing student writing 
assessing other skills 
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OUgSTTGM 2 

The two basic aims of the Primary Assessment Progra^n are: 

tomonltor standards of achievement in English and Machemacics 
in Northern Territory primary schools. 

to provide teachers in the upper "primary with materials for 
assessing the competency of students in English and Mathematics. 

Do you consider that these aims are being met? 

Aim i 



Aim 2 



QU5STIQH 3 

Please connent on how you use the Primary Assessnent materials in your 
school and v<t^at value they are to the school? 



What do you see as " the strengths and weaknesses cf :r.e Pri.nary 
Assessment Prcgram? 

Strengths 



is; 7. 



Weaknesses 
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QUSSTIQN 5 

Hew did you find out aoout the program? 



QUKSTTOM 6 

Do you meet with your staif to discuss the program? QYes ( [ No 
If yes, what sorts of things are discussed? 



Do you think the assessment program is he 
standards across the Territory? Q"^" 



ping to Improve academic 
No Please ccnment. 



CU^STTCM B 

Hew do you feel aisout the program as a who If? 
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GUZ5TICMHAi:;£ ?C.9 mCHESS OF U??g5 Por^APV CLASSES - vZ^TiS 5. a AND 7 

Have ycu ever seen "hese material a? Please answer Iss or Hfl in the boxes 
provided. Please indicate dy ticking <y) the dcxes these materials you 
have used this v<»Ar or have used in the oagt. ■ 

Please Indicate using A. 3, C, or D, what each of the materials is/uas 
used for . (Note: they may have more than one use) 



A - system-wide tests selected from the Item pools 

B - your own tests or class exercises selected frcin the item pools 

C - assessing student writing 

D - assessing other skills 



ERIC 




QUEST! CN 2 

What do you see as the strengths and weaknesses of: 

The Mathematlcgand Reading Itgm Pools ; 

Strengths 



Weaknesses 



Thg Writing Moderation Booklets 
Strengths 



Weaknesses 



Type C ' St rategies for Aggessing Reading 
Strengths 



Weaknesses. 
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GUE5TTCH 3 

Hew did you find out ancut the Primary Assessment Program? 



QUESTIQN 4 

Do you meet with other staff to discuss the program? [jYes | [no 
If yes, what sorts of things are discussed? 



QOESTTOH 5 

Do you think- the assessment program Is helping to Improve academic 
standards across the Territory? Q^es Please carmen t. 



QUgSTTGW 6 

How do you feci about the program as a whole? 



mSSTTCN 7 

Are ycu aware of the changes made to ^'he prograin In 1988? []j':fes QjfJo 
If yes please ccmment on the changes. 

Mathematics « • 



iJeadlng. 
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CUESTICN 8 



Have ycu seen the test results chat have been sent to ycur scnccl? 
QjYes I \tio It yes, in what ways Co you use chem? 



QUESTION 9 

Have you read the reports on the results that have been Issued each 
year? [ [Yes [ [No If yes, please cement on their usefulness. 



What do you consider to be the most suitalDle time of the school year for 
administering the system-wide tests? 



Mathematics. 
Reading 



QUESTION tl 

Any other comments? (e.g. on the materials, test administration etc.) 



QUESTION 12 

Year level taug^it/posi tion In school? 
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QUESTTONNAIPg: rOP ?prHCI?ALS/ASSI37AOT POIMCIPALS Agc??TGIMAL SCHOOLS 



QUESTTQH t 

The materials shewn belcw have been sent to your school over the pas; 
three years. Have you seen these materials? Please answer isa or Jla in 
the boxes prov-ided.- Please indicate by ticking (/) the boxes those 
materials that have been used thig vear or have been. used in the o^sz . 
Please Indicate using A, B, C, or D» what each of the raacerials is/was 
used for . (Motet- thev may have more- than one use) - • " 

A - system-wide tests selected from the item pools 

B - class tests or exercises selected frca the item pools 

C - assessing student writing 

D - assessing other skills 
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MATERIALS 



MATSaiALS 
SS2I Yes/No 



US2D THIS 
YIA3 U) 



.MATHE71AT1C3 




Toldmv IB 9 6 



READfffG 

rraipcoL 




US2D IN Ta£ 
PAST cy) 



USED :GH 
A. 3, CD 




9^ 



[!ESTCOPYJiVAILIBLE 



GU^STTCH ? 

The tvo basic aims cf the Primary Assessment Program are: 

to monitor standards of achievement In English and Mathematics 
in Northern Territory primary, schcojs, 

to provide teachers in the upper primary with materials for 
. . ..assessing the ccmpetency. of. students in. English and Mathematics. - -- 

Do you consider that these aims are being met? 

Aim 1 



Aim 2 



QUESTION 3 

Please comment on how you use the Primary Assessment materials in your 
school and what value they are to the school? 



What do you see as the strengths and weaknesses of the Primary 
Assessment Program? 

Strengths 
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GUESTTCM 5 

Hew did ycu find cut ai2cut the program? 



QU-ESTTOM S 

Do you meet with ycur staff to discuaa the program? 0^23 I I No 
If yea, what sorts of things are discussed? — 



GtTgSTTOM 7 



Do you think the assessment program is 
standards across the Territory? Qj' 



:e3 



he ping 

Quo 



to Improve acacsaic 
Please ccrrsgnt. 



Kcw rto ycu isel aficut the prcgrar. as a whole'' 



Any other ccnr:ents? (e,g. on the raaterials, test acninistracicn e:c.) 



CUESTTGM 10 

Are ycu aware of the changes Mde tc the prcgraa In 1988. QjYea | [no 
Please ccaaient on the changes. 



Have ycu seen the test results that have teen sent to ycur school? 
[ [Yes I [Ho What use dees the school make of the results? 



Have ycu read the reports on the results that have been Issued each 
year? Q Cj^^ please ccnasnt cn their usefulness. 



Vhat do ycu consider to be the scst 5uita£:le time of the scnccl year :cr 
acainlscer ing the systea-wlde tests? 
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Have you ever seen these raaterUla? Please answer ilsa cr ilfl In sne acxes 
provided.- Please indicate 6y ticking <✓) the boxes those materials -ycu 
have used this year nr h ^ve us»d in the Q;».«sg . 

Please indicate using A, B, C, or- D, what each of the'materlaVs is/was 
uagti ^Qr,' (Mote: they may have more than one use) 



A 
B 
C 
D 



system-vide tests selected frcm the item pools 
your own tests or class exercises selected frcm the Item pools 
assessing student writing- 
assessing other skills 
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MATERIALS 



MATHHMATICa 



Fold«r 19 8« 



MATERIALS 
SEES Yes/No 



USED THIS 
YZAi? <y) 



?old«r. L986 
y«lIov covar 



R£ADHSG 
nSMPOOL 




3QOjci«c 19 as 



?oii«r 1936 
driJi^« cava: 




■ f £4C.tV<3 ' 



US2D IN THE 
PAST W) 



US&U FOR 

A, a, CO 
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BES^CflPYAYMfJLE 



QtJSSTION 2 

What ao you see as the strengths ana weaknesses of; 

The Ma thema i: 1 cs and - gead i na 1 1 em ?oo 1 s_ 

Strengths 



Weaknesses 



Thg Writing Moderation Booklets 
Strengths 



Weaknesses 



Type C - StratgQies for As3e *tgtna geadina 
Strengths 



Weaknesses 



do • 

o 



GUESTIQH 3 

How did you find out aocut the Primary Assessaent Program? 
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Do you meet with other etaff to discuss the program? riYes [~]No 
If yest what sorts of things are discussed? 



QUESTIQH 5 

Do you think the assessment program Is helping to Improve academic 
standards across the Territory? Q^^^ Q^*^ Please comment. 



QUESTION 6 

Hew do you feel atcut the program as a whole? 



GUg:STTQN 7 

Ar^f ycu aware of the changes made to the program in 1988? Qj'J^ss |^ [No 

If yes please ccianent on the changes. 

Mathematics • 



Reading. 

'dB 

_flr3 



GUEST I ON a 



Have you seen the test results that have Deen sent to your school? 
' lH^^ ^' ^**^' ^"^^ them? 



QUESTION 9 

Have you read the reports on the results that have been issued each 
year? Q^es QjNo please cooment on their usefulness. 



nm:STTOH 10 

What do you consider to be the most suitable time oi the school year for 
adnlnistering the system-wide tests? 

Mathematics. • • • • • • * * * 

Readiiig ..I..., : 

fJUFSTTON 11 

Any other ccnsnents? <e.g. on the materials, test administration etc.) 



QU?:5TT0N 12 

Year level taught/poai tlon In school? 



iOu, 
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Appendix vii 



EVALUATION OF THE PRIMARY ASSESSMENT PROGRAM 
1988 Year 7 MATHEMATICS TEST. 



Ploase comment in the spaces provided or on the reverse side of this sheet. 



SCHCOL: 



1 . TEST ADMINISTRATION: (e.g. time allowed for Uie test, data of 
administration, test format ...) 



2. TEST: (e.g. range and difficulty of items, relationship to the curriculum, ralavanca 
to currant teaching practice ...) 



3. FURTHER COMMENTS regarding this test and Primar/ Assessment Program in 
gsnaral: 
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